Cloud vs Cloud vs Physical - Benchmarking cloud & server performance with PerfKitBenchmarker

posted Jul 30, 2016, 2:27 PM by William Shallum [ updated Jul 30, 2016, 2:51 PM ]

These days many companies provide cloud services. From the big players like Amazon (AWS), Google (GCP), and Microsoft (Azure), to the smaller ones, to the traditional hosting companies. The multitude of competing offers and pricing models can be confusing. While I don't normally deal with the financial side, I am sometimes expected to help on the technical side.

The good folks at Google along with some other companies and academic institutions have created a tool called PerfKitBenchmarker. It is written in Python and uses the cloud vendor-provided CLI tools to provision instances, run benchmarks on them, and (most importantly) terminate them afterwards[*]. Here I will show how to configure and run some simple tests using PerfKitBenchmarker (let's call it PKB from here on). I assume you use Linux or something similar. These steps are tested on CentOS 7. The installation part is mostly just me restating the documentation, but maybe you will enjoy the example configuration?

Installation

Installing PKB is mostly like installing any other python (2.7, not 3) package (except that it's not on PyPI), start with a virtualenv:

$ virtualenv-2.7 ~/pkb-ve
New python executable in /home/you/pkb-ve/bin/python
Installing Setuptools...done.
Installing Pip...done.

$ . ~/pkb-ve/bin/activate
(pkb-ve)$ pip install -U pip
Downloading/unpacking pip from https://pypi.python.org/packages/[...snip...]/pip-8.1.2.tar.gz#md5=[...snip...]
Downloading pip-8.1.2.tar.gz (1.1MB): 1.1MB downloaded
Running setup.py egg_info for package pip

    warning: no previously-included files found matching [...snip...]
    no previously-included directories found matching [...snip...]
Installing collected packages: pip
Found existing installation: pip 1.4.1
    Uninstalling pip:
      Successfully uninstalled pip
Running setup.py install for pip

warning: no previously-included files found matching [...snip...]
no previously-included directories found matching [...snip...]

    Installing pip script to /home/you/pkb-ve/bin
    Installing pip2.7 script to /home/you/pkb-ve/bin
    Installing pip2 script to /home/you/pkb-ve/bin
Successfully installed pip
Cleaning up...
(pkb-ve)$

The pip upgrade is mostly so we can use precompiled binaries (wheels) for some requirements.

Download the latest release of PKB from Github: https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/releases and untar it somewhere, for this example ~/pkb.

Now install the requirements:

(pkb-ve)$ cd ~/pkb

(pkb-ve)$ pip install -r requirements.txt
Collecting python-gflags==3.0.4 (from -r requirements.txt (line 14))
Using cached python-gflags-3.0.4.tar.gz

[...snip...]

Successfully installed MarkupSafe-0.23 PyYAML-3.11 blinker-1.4 colorama-0.3.7 colorlog-2.6.0 futures-3.0.5 jinja2-2.8 numpy-1.11.1 pandas-0.18.1 pint-0.7.2 python-dateutil-2.5.3 python-gflags-3.0.4 pytz-2016.6.1 six-1.10.0

(pkb-ve)$

As stated above, PKB uses vendor tools. Here we will use Azure because I got some credits for free from the Visual Studio Dev Essentials Program[**].

The Azure tools are (sadly?) written in Javascript and use node.js - so you can't just reuse the virtualenv from earlier. If you don't already have node.js, packages are available from EPEL and also from SCL. In this example I am using the EPEL package (version 0.10).

If you're OK with messing up your system you can of course sudo npm install azure-cli@0.9.9 -g but let's just mess up one part of our system at a time.

(pkb-ve)$ cd ~/pkb-ve
(pkb-ve)$ npm install azure-cli@0.9.9

[...snip downloading the Internet...]
azure-cli@0.9.9 node_modules/azure-cli

[...snip dependency tree...]

(pkb-ve)$ ln -s ~/pkb-ve/node_modules/.bin/azure ~/pkb-ve/bin

(pkb-ve)$ azure
info:             _    _____   _ ___ ___
info:            /_\ |_ / | | | _ \ __|
info:      _ ___/ _ \__/ /| |_| |   / _|___ _ _
info:    (___ /_/ \_\/___|\___/|_|_\___| _____)
info:       (_______ _ _)         _ ______ _)_ _
info:              (______________ _ )   (___ _ _)
info:
info:    Microsoft Azure: Microsoft's Cloud Platform

[...snip help...]
(pkb-ve)$

(the "azure" text in the ASCII art really is blue).

So we have installed azure-cli to a node_modules directory inside the virtualenv, and created a symlink to the CLI utility in the virtualenv's bin directory which is in the path when the virtualenv is active. PKB expects to find the CLI in the path.

Now configure the CLI to use your account. Just follow the instructions given by the Azure CLI:

(pkb-ve)$ azure account download
info:    Executing command account download
info:    Launching browser to http://go.microsoft.com/fwlink/?LinkId=254432
help:    Save the downloaded file, then execute the command
help:      account import <file>
info:    account download command OK
(pkb-ve)$ azure account import ~/Downloads/blablabla.publishsettings
info:    Executing command account import
info:    account import command OK
(pkb-ve)$

Now test if it works:

(pkb-ve)$ azure vm list

+ Getting virtual machines

info: No VMs found
info: vm list command OK

(pkb-ve)$

Once the account is configured, we can move on to configuring the benchmark(s).

Configuration

PKB has many benchmarks but in this case let's run a simple cross-region iperf test. PKB configuration uses YAML so create this file, let's call it iperf.yaml:

small_sea: &small_sea
Azure:
    machine_type: Small
    zone: Southeast Asia

small_us: &small_us
Azure:
    machine_type: Small
    zone: West US

iperf_azure: &iperf_azure
flags:
    ip_addresses: EXTERNAL
vm_groups:
    vm_1:
      cloud: Azure
      vm_spec: *small_us
    vm_2:
      cloud: Azure
      vm_spec: *small_sea

benchmarks:
- iperf: *iperf_azure

The top-level keys that matter are (I think...) benchmarks and benchmark_name (e.g. iperf, ...). Other keys are ignored so they are ideal to declare anchors we can reference later. If you want to see the configuration with the references expanded try putting it into http://www.yamllint.com/.

Now we can run PKB using this file as configuration (interesting parts in bold):

(pkb-ve)$ cd ~/pkb
(pkb-ve)$ ./pkb.py --benchmark_config_file=iperf.yaml

2016-07-31 02:53:10,071 e1ddd081 MainThread INFO     Verbose logging to: /tmp/perfkitbenchmarker/runs/e1ddd081/pkb.log

2016-07-31 02:53:10,072 e1ddd081 MainThread INFO PerfKitBenchmarker version: unknown
2016-07-31 02:53:10,072 e1ddd081 MainThread INFO Flag values:
--benchmark_config_file=iperf.yaml

2016-07-31 02:53:10,308 e1ddd081 MainThread WARNING  The key "cloud" was not in the default config, but was in user overrides. This may indicate a typo.

2016-07-31 02:53:10,308 e1ddd081 MainThread WARNING  The key "cloud" was not in the default config, but was in user overrides. This may indicate a typo.

2016-07-31 02:53:10,309 e1ddd081 MainThread WARNING  The key "flags" was not in the default config, but was in user overrides. This may indicate a typo.

2016-07-31 02:53:10,347 e1ddd081 MainThread INFO Running: azure -v
2016-07-31 02:53:10,600 e1ddd081 MainThread iperf(1/1) INFO Provisioning resources for benchmark iperf

2016-07-31 02:53:10,606 e1ddd081 Thread-3 iperf(1/1) INFO     Running: azure account affinity-group create --location=Southeast Asia --label=pkbe1ddd081c5e55b7222d7 pkbe1ddd081c5e55b7222d7

[...snip...]

2016-07-31 02:54:03,702 e1ddd081 Thread-2 iperf(1/1) INFO     Ran azure network vnet create --affinity-group=pkbe1ddd0816d400dff5650 pkbe1ddd0816d400dff5650. Got return code (1).

[...snip...]
STDERR: error: An update to your network configuration is currently underway. Please try this operation again later.
error: network vnet create command failed

[...snip...]
2016-07-31 02:59:11,406 e1ddd081 MainThread iperf(1/1) INFO ssh to VMs in this benchmark by name with:
ssh -F /tmp/perfkitbenchmarker/runs/e1ddd081/ssh_config <vm_name>
ssh -F /tmp/perfkitbenchmarker/runs/e1ddd081/ssh_config vm<index>
ssh -F /tmp/perfkitbenchmarker/runs/e1ddd081/ssh_config <group_name>-<index>
2016-07-31 02:59:11,411 e1ddd081 MainThread iperf(1/1) INFO Preparing benchmark iperf

[...snip...]

2016-07-31 03:05:49,850 e1ddd081 MainThread iperf(1/1) INFO     Ran ssh -A -p 22 perfkit@13.76.140.92 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /tmp/perfkitbenchmarker/runs/e1ddd081/perfkitbenchmarker_keyfile iperf --client 13.93.227.187 --port 20000 --format m --time 60 -P 1. Got return code (0).

STDOUT: ------------------------------------------------------------
Client connecting to 13.93.227.187, TCP port 20000
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[ 3] local 10.32.0.4 port 40142 connected with 13.93.227.187 port 20000
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 969 MBytes 135 Mbits/sec

STDERR: Warning: Permanently added '13.76.140.92' (ECDSA) to the list of known hosts.

2016-07-31 03:05:49,855 e1ddd081 MainThread iperf(1/1) INFO     Cleaning up benchmark iperf
2016-07-31 03:05:49,855 e1ddd081 MainThread iperf(1/1) INFO     Tearing down resources for benchmark iperf
2016-07-31 03:05:49,857 e1ddd081 Thread-165 iperf(1/1) INFO     Running: azure vm delete --quiet pkb-e1ddd081-0

[...snip...]
2016-07-31 03:11:02,215 e1ddd081 MainThread INFO
-------------------------PerfKitBenchmarker Complete Results-------------------------

[...snip...]
-------------------------PerfKitBenchmarker Results Summary-------------------------
IPERF:
ip_type="external" receiving_machine_type="Small" runtime_in_seconds="60" sending_machine_type="Small" sending_thread_count="1"
Throughput                          133.000000 Mbits/sec                      (receiving_zone="Southeast Asia" sending_zone="West US")
Throughput                          135.000000 Mbits/sec                      (receiving_zone="West US" sending_zone="Southeast Asia")
End to End Runtime                 1071.605115 seconds

-------------------------

For all tests: perfkitbenchmarker_version="unknown" vm_1_cloud="Azure" vm_1_image="b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_4-LTS-amd64-server-20160714-en-us-30GB" vm_1_machine_type="Small" vm_1_vm_count="1" vm_1_zone="West US" vm_2_cloud="Azure" vm_2_image="b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_4-LTS-amd64-server-20160714-en-us-30GB" vm_2_machine_type="Small" vm_2_vm_count="1" vm_2_zone="Southeast Asia"

2016-07-31 03:11:02,216 e1ddd081 MainThread INFO     Publishing 3 samples to /tmp/perfkitbenchmarker/runs/e1ddd081/perfkitbenchmarker_results.json

2016-07-31 03:11:02,216 e1ddd081 MainThread INFO Benchmark run statuses:
------------------------
Name UID Status
------------------------
iperf iperf0 SUCCEEDED
------------------------
Success rate: 100.00% (1/1)

2016-07-31 03:11:02,217 e1ddd081 MainThread INFO     Complete logs can be found at: /tmp/perfkitbenchmarker/runs/e1ddd081/pkb.log

(pkb-ve)$

Just ignore the scary WARNING parts. The cloud and the flags are usually specified on the command line (the same vm_spec can have configurations for multiple clouds). I'm not sure I like troubleshooting the cascading flags so I prefer to just configure everything in the config file[***].

Since operations in "the Cloud" take some time to respond, some errors are possible (e.g. see above). PKB will retry the operations after a slight delay.

As you can see, it takes a long time to finish, but at least you didn't have to click on anything and it cleans up after itself.

Configuration - Static VM

Contrary to popular belief, not everyone has moved to "the Cloud" yet. You may have some machines in your own datacenter or on some hosting provider that is insufficiently "cloudy" and is unsupported by PKB. Fear not! You can run PKB on those machines by using the static_vms feature ("static" probably means PKB will not provision them?). Here we will run the same iperf benchmark as above. This will connect to a VPS and run iperf between it and an Azure VM. Let's call this file iperf-static.yaml:

small_sea: &small_sea
Azure:
    machine_type: Small
    zone: Southeast Asia

small_us: &small_us
Azure:
    machine_type: Small
    zone: West US

static_vms:
us-vps: &us-vps
    ip_address: your.ip.here
    user_name: your_user
    ssh_private_key: /path/to/ssh/key/here # hopefully passwordless?
    ssh_port: 22

os_type: debian # It's actually Alpine Linux, not Debian, but...

    install_packages: False # I installed iperf and configured the firewall before running PKB.

iperf_azure_vps: &iperf_azure_vps
flags:
    ip_addresses: EXTERNAL
vm_groups:
    vm_1:
      cloud: Azure
      vm_spec: *small_us
    vm_2:
      static_vms:
        - *us-vps

benchmarks:
- iperf: *iperf_azure_vps

Run it like before (relevant parts and commands run on static VM highlighted):

$ ./pkb.py --benchmark_config_file=iperf-static.yaml

2016-07-31 03:46:31,497 558636a2 MainThread INFO     Verbose logging to: /tmp/perfkitbenchmarker/runs/558636a2/pkb.log

2016-07-31 03:46:31,497 558636a2 MainThread INFO PerfKitBenchmarker version: unknown
2016-07-31 03:46:31,497 558636a2 MainThread INFO Flag values:
--benchmark_config_file=iperf-static.yaml

2016-07-31 03:46:31,667 558636a2 MainThread WARNING  The key "static_vms" was not in the default config, but was in user overrides. This may indicate a typo.

2016-07-31 03:46:31,668 558636a2 MainThread WARNING  The key "cloud" was not in the default config, but was in user overrides. This may indicate a typo.

2016-07-31 03:46:31,668 558636a2 MainThread WARNING  The key "flags" was not in the default config, but was in user overrides. This may indicate a typo.

2016-07-31 03:46:31,708 558636a2 MainThread INFO Running: azure -v
2016-07-31 03:46:31,925 558636a2 MainThread iperf(1/1) INFO Provisioning resources for benchmark iperf

[...snip...]

2016-07-31 03:47:59,046 558636a2 Thread-9 iperf(1/1) INFO VM: your.ip.here

2016-07-31 03:47:59,048 558636a2 Thread-9 iperf(1/1) INFO     Running: ssh -A -p 22 your_user@your.ip.here -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /path/to/ssh/key/here hostname

2016-07-31 03:48:01,412 558636a2 Thread-9 iperf(1/1) INFO     Running: ssh -A -p 22 your_user@your.ip.here -2 -o UserKnownHostsFile=/dev/null 
-o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o 
PreferredAuthentications=publickey -o PasswordAuthentication=no -o 
ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o
 ServerAliveCountMax=10 -i /path/to/ssh/key/here mkdir -p /tmp/pkb

[...snip...]
2016-07-31 03:51:25,663 558636a2 MainThread iperf(1/1) INFO Preparing benchmark iperf

[...snip...]

2016-07-31 03:54:54,948 558636a2 MainThread iperf(1/1) INFO     Running: ssh -A -p 22 perfkit@137.135.47.113 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /tmp/perfkitbenchmarker/runs/558636a2/perfkitbenchmarker_keyfile nohup iperf --server --port 20000 &> /dev/null& echo $!

2016-07-31 03:54:58,539 558636a2 MainThread iperf(1/1) INFO     Running: ssh -A -p 22 your_user@your.ip.here -2 -o UserKnownHostsFile=/dev/null 
-o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o 
PreferredAuthentications=publickey -o PasswordAuthentication=no -o 
ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o
 ServerAliveCountMax=10 -i /path/to/ssh/key/here nohup iperf --server --port 20000 &> /dev/null& echo $!

2016-07-31 03:55:00,982 558636a2 MainThread iperf(1/1) INFO Running benchmark iperf
2016-07-31 03:55:00,985 558636a2 MainThread iperf(1/1) INFO Iperf Results:

2016-07-31 03:55:00,986 558636a2 MainThread iperf(1/1) INFO     Running: ssh -A -p 22 perfkit@137.135.47.113 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /tmp/perfkitbenchmarker/runs/558636a2/perfkitbenchmarker_keyfile iperf --client your.ip.here --port 20000 --format m --time 60 -P 1

2016-07-31 03:56:05,141 558636a2 MainThread iperf(1/1) INFO     Ran ssh -A -p 22 perfkit@137.135.47.113 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /tmp/perfkitbenchmarker/runs/558636a2/perfkitbenchmarker_keyfile iperf --client your.ip.here --port 20000 --format m --time 60 -P 1. Got return code (0).

STDOUT: ------------------------------------------------------------
Client connecting to your.ip.here, TCP port 20000
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[ 3] local 10.32.0.4 port 42062 connected with your.ip.here port 20000
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 3576 MBytes 500 Mbits/sec

STDERR: Warning: Permanently added '137.135.47.113' (ECDSA) to the list of known hosts.

2016-07-31 03:56:05,144 558636a2 MainThread iperf(1/1) INFO     Running: ssh -A -p 22 your_user@your.ip.here -2 -o UserKnownHostsFile=/dev/null 
-o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o 
PreferredAuthentications=publickey -o PasswordAuthentication=no -o 
ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o
 ServerAliveCountMax=10 -i /path/to/ssh/key/here iperf --client 137.135.47.113 --port 20000 --format m --time 60 -P 1

2016-07-31 03:57:18,261 558636a2 MainThread iperf(1/1) INFO     Ran ssh -A -p 22 your_user@your.ip.here -2 -o UserKnownHostsFile=/dev/null 
-o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o 
PreferredAuthentications=publickey -o PasswordAuthentication=no -o 
ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o
 ServerAliveCountMax=10 -i /path/to/ssh/key/here iperf --client 137.135.47.113 --port 20000 --format m --time 60 -P 1. Got return code (0).

STDOUT: ------------------------------------------------------------
Client connecting to 137.135.47.113, TCP port 20000
TCP window size: 0.04 MByte (default)
------------------------------------------------------------
[ 3] local your.ip.here port 51766 connected with 137.135.47.113 port 20000
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 7949 MBytes 1111 Mbits/sec

STDERR: Warning: Permanently added 'your.ip.here' (ECDSA) to the list of known hosts.

2016-07-31 03:57:18,266 558636a2 MainThread iperf(1/1) INFO Cleaning up benchmark iperf

2016-07-31 03:57:18,266 558636a2 MainThread iperf(1/1) INFO     Running: ssh -A -p 22 perfkit@137.135.47.113 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /tmp/perfkitbenchmarker/runs/558636a2/perfkitbenchmarker_keyfile kill -9 2361

2016-07-31 03:57:22,211 558636a2 MainThread iperf(1/1) INFO     Running: ssh -A -p 22 your_user@your.ip.here -2 -o UserKnownHostsFile=/dev/null 
-o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o 
PreferredAuthentications=publickey -o PasswordAuthentication=no -o 
ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o
 ServerAliveCountMax=10 -i /path/to/ssh/key/here kill -9 2472

2016-07-31 03:57:28,819 558636a2 MainThread iperf(1/1) INFO Tearing down resources for benchmark iperf

[...snip...]
2016-07-31 04:01:11,247 558636a2 MainThread INFO
-------------------------PerfKitBenchmarker Complete Results-------------------------

[...snip...]
-------------------------PerfKitBenchmarker Results Summary-------------------------
IPERF:
ip_type="external" runtime_in_seconds="60" sending_thread_count="1"
Throughput                          500.000000 Mbits/sec                      (receiving_machine_type="None" receiving_zone="Static - your_user@your.ip.here" sending_machine_type="Small" sending_zone="West US")
Throughput                         1111.000000 Mbits/sec                      (receiving_machine_type="Small" receiving_zone="West US" sending_machine_type="None" sending_zone="Static - your_user@your.ip.here")
End to End Runtime                  879.311017 seconds

-------------------------

For all tests: perfkitbenchmarker_version="unknown" vm_1_cloud="Azure" vm_1_image="b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_4-LTS-amd64-server-20160714-en-us-30GB" vm_1_machine_type="Small" vm_1_vm_count="1" vm_1_zone="West US" vm_2_cloud="Static" vm_2_image="None" vm_2_vm_count="1" vm_2_zone="Static - your_user@your.ip.here"

2016-07-31 04:01:11,249 558636a2 MainThread INFO     Publishing 3 samples to /tmp/perfkitbenchmarker/runs/558636a2/perfkitbenchmarker_results.json

2016-07-31 04:01:11,250 558636a2 MainThread INFO Benchmark run statuses:
------------------------
Name UID Status
------------------------
iperf iperf0 SUCCEEDED
------------------------
Success rate: 100.00% (1/1)

2016-07-31 04:01:11,251 558636a2 MainThread INFO     Complete logs can be found at: /tmp/perfkitbenchmarker/runs/558636a2/pkb.log

(pkb-ve)$

The process is similar except that the static VM is neither provisioned nor cleaned up. If install_packages is not False (and maybe even if it is true?), the user must have passwordless sudo access. Fortunately the iperf benchmark does not need any privileged commands.

Conclusion

If your office network is still using Fast Ethernet, it has less bandwidth than a link between West US and Southeast Asia on the Internet.

For more information e.g. about configuration for the individual benchmarks, the actions each benchmark takes, please look at the source code.

References

PerfKitBenchmarker: https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/
PerfKitBenchmarker linux benchmarks (to look at configurations, etc): https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/tree/master/perfkitbenchmarker/linux_benchmarks
Fedora EPEL: https://fedoraproject.org/wiki/EPEL
Software Collections node.js 4: https://www.softwarecollections.org/en/scls/rhscl/rh-nodejs4/
Azure Portal: https://portal.azure.com/
Visual Studio Dev Essentials (free Azure credits - no credit card required): https://www.visualstudio.com/en-us/products/visual-studio-dev-essentials-vs.aspx

[*] Terminating them is important if you like your money and wish to keep it.

[**] Which is pretty cool. I knew paying for Windows 7 Pro would pay off one day.

[***] From the documentation (emphasis added):

--zones: A list of zones within which to run PerfKitBenchmarker. This is
    specific to the cloud provider you are running on. If multiple zones are
    given, PerfKitBenchmarker will create 1 VM in zone, until enough VMs are
    created as specified in each benchmark. The order in which this flag is
    applied to VMs is undefined.

Comments

William Shallum

Quick Links

Notice