Random Notes

Random useful(?) notes.
Subscribe to posts

Random SSH Agent Tricks

posted Oct 26, 2017, 8:58 AM by William Shallum   [ updated Oct 26, 2017, 8:59 AM ]

If you have a Linux Server somewhere, you typically access it using SSH. If your server is accessible over the Internet, the log files will tell you that it receives hundreds to thousands of brute force login attempts per day.

Sep 10 04:19:59 vm auth.info sshd[8360]: Invalid user ubnt from 1.54.xx.yy
Sep 10 04:19:59 vm auth.info sshd[8360]: input_userauth_request: invalid user ubnt [preauth]
Sep 10 04:19:59 vm auth.err sshd[8360]: error: Received disconnect from 1.54.xx.yy port 57526:3: com.jcraft.jsch.JSchException: Auth fail [preauth]
Sep 10 04:19:59 vm auth.info sshd[8360]: Disconnected from 1.54.xx.yy port 57526 [preauth]
Sep 10 04:20:19 vm auth.info sshd[8362]: Invalid user support from 1.54.xx.yy
Sep 10 04:20:19 vm auth.info sshd[8362]: input_userauth_request: invalid user support [preauth]
Sep 10 04:20:19 vm auth.err sshd[8362]: error: Received disconnect from 1.54.xx.yy port 58807:3: com.jcraft.jsch.JSchException: Auth fail [preauth]

This means "12345" is not a good root password. Actually you probably want to log in using a key instead, as that's a lot harder to brute force than "12345". Feel free[1] to use "12345" for your local private key password if you miss it.

So, like any security-minded individual you protect your private keys on disk with a password. The problem with this is you have to type the password to decrypt the key every time you want to use it. The ssh-agent is a program that will stay in memory, and hold the decrypted private key for you, so it can be used by your SSH processes.

So everything is OK, you enter the password once, and the agent has the decrypted key. Your future ssh sessions automatically contact the agent to get the keys.

1. Do not use the agent from GNOME Keyring

user@vm:~$ ssh-add -l
2048 30:cb:fb:28:09:69:da:22:3a:12:4a:0a:8a:03:3b:f0 user@vm (RSA)
user@vm:~$ ssh-add -D
All identities removed.
user@vm:~$ ssh-add -l
2048 30:cb:fb:28:09:69:da:22:3a:12:4a:0a:8a:03:3b:f0 user@vm (RSA)
user@vm:~$ ssh-add ~/.ssh/id_ecdsa
Error reading response length from authentication socket.
Could not add identity: /home/user/.ssh/id_ecdsa
user@vm:~$ ssh-add ~/.ssh/id_ed25519
Error reading response length from authentication socket.
Could not add identity: /home/user/.ssh/id_ed25519
user@vm:~$ echo $SSH_AUTH_SOCK
/run/user/1000/keyring-aUXdpV/ssh

The agent from GNOME keyring cannot remove a passwordless default key, it cannot handle key types other than DSA & RSA, it also does not support the tricks here. On Ubuntu 14.04 with Unity, run gnome-session-properties, uncheck the SSH Agent, and restart the desktop session.

2. Add a lifetime to the keys in your agent

Ideally you would remove keys after you finish using them. This reduces the possibility that someone else will use them. If you often don't remember to remove the keys, why not let the computer do it for you?

user@vm:~$ ssh-add -t 30 ~/.ssh/id_rsa; ssh-add -l ; sleep 31; ssh-add -l
Identity added: /home/user/.ssh/id_rsa (/home/user/.ssh/id_rsa)
Lifetime set to 30 seconds
2048 30:cb:fb:28:09:69:da:22:3a:12:4a:0a:8a:03:3b:f0 /home/user/.ssh/id_rsa (RSA)
The agent has no identities.

By using the -t option, keys will be automatically removed after the time has passed. The time unit is seconds by default, but you can also use "10m" (minutes), "10h" (hours), "10d" (days) and combinations like "1h30m" or "10d10h10m10s"[2].

3. Prompt for confirmation when the key is used

Let's say you have your keys in the agent and ready to use. How do you know that you are the only one using them? Anyone who logs in as root or the same user will be able to connect to the agent's socket and request keys. One way is to make the agent prompt for confirmation for each use with the -c option. This probably works better with agent forwarding as local root can probably do other, nastier things to the agent process.

user@vm:~/.ssh$ ssh-add -c ~/.ssh/id_ecdsa.priv
Identity added: /home/user/.ssh/id_ecdsa.priv (/home/user/.ssh/id_ecdsa.priv)
The user must confirm each use of the key

Afterwards, if you try to use it, a box like this pops up:

SSH Askpass prompt
The askpass prompt for confirmation. The textbox does nothing, just click OK or Cancel.

If you press OK, the agent allows use of the key and the connection attempt succeeds. If you press Cancel, the agent will reject the request as seen below:

user@vm:~/.ssh$ ssh localhost
Agent admitted failure to sign using the key.
user@localhost's password:

4. Lock your agent

The ssh-agent can be locked with a password. The agent must be unlocked for the keys inside to be usable again.

user@vm:~$ ssh-add
Enter passphrase for /home/user/.ssh/id_rsa:
Identity added: /home/user/.ssh/id_rsa (/home/user/.ssh/id_rsa)
user@vm:~$ ssh-add -l
4096 00:11:22:33:44:55:66:77:88:99:aa:bb:cc:dd:ee:ff /home/user/.ssh/id_rsa (RSA)
user@vm:~$ ssh-add -x
Enter lock password:
Again:
Agent locked.
user@vm:~$ ssh-add -l
The agent has no identities.
user@vm:~$ ssh-add -X
Enter lock password:
Agent unlocked.
user@vm:~$ ssh-add -l
4096 00:11:22:33:44:55:66:77:88:99:aa:bb:cc:dd:ee:ff /home/user/.ssh/id_rsa (RSA)




[1]: don't do this.
[2]: See sshd_config manual page, TIME FORMATS section.

Random Ansible "Tricks"

posted Nov 12, 2016, 1:08 AM by William Shallum   [ updated Nov 12, 2016, 1:09 AM ]

Ansible is a thing used at work, it is sometimes rather annoying, but at least it's somewhat easy to debug. Here are some things I found during experimenting.

No such thing as async include

The unit of reuse is the yml file. What I would like to do is something like this pseudocode:

jobs = []
for (i = 1 to 10) {
  jobs += async {
    do_some(i)
    long_running(i)
    tasks(i)
  }
}
wait_for_all(jobs)

async applies only to a task. Include is not a task, therefore you cannot do this:


- include: somefile.yml i={{ item }}
  with_items: 
    - 1
    - 2
  async: 3600
  poll: 0
  register: jobs
- async_status: jid=item.ansible_job_id
  register: job
  until: job.finished
  retries: 3000
  with_items: "{{ jobs.results }}"

so as a workaround, if there is only one long running task in the included file:


- set_fact:
    jobs: []
- include: somefile.yml i={{ item }}
  with_items: "{{ items }}"
- async_status: jid=item.ansible_job_id
  register: job
  until: job.finished
  retries: 3000
  with_items: "{{ jobs }}"

somefile.yml:

- quick_pre_task: i={{ i }}
- long_running_task: i={{ i }}
  async: 3600
  poll: 0
  register: long_running
- set_fact:
    jobs: "{{ jobs|default([]) + [long_running] }}

This is equivalent to this pseudocode (i.e. quick_pre_task is still run synchronously and serially):

jobs = []
for (i = 1 to 10) {
  quick_pre_task(i)
  jobs += async { long_running_task(i) }
}
wait_for_all(jobs)

Ugh. Other workarounds may include just running another ansible-playbook command as async - say goodbye to your context.

Using Jinja2 control structures in the playbook

So let's say you want the private IP address, no matter what interface it is on, as a parameter. Normally to use the value of a variable you would do: 

- sometask: someparam={{ somevar }}

{{ somevar }} is of course evaluated using Jinja2 resulting in the value of the variable being substituted in. All very good, but the Jinja2 engine is used in its full glory here, so you can actually do:

- sometask: someparam="{% for ip in ansible_all_ipv4_addresses if ip.startswith('10.') %}{% if loop.first %}{{ ip }}{% endif %}{% endfor %}"

People will probably hate you for that.

As a sidenote, it seems that the basic structure (command option=argument) is parsed first before argument is passed to Jinja2 so you cannot do:

- set_fact:
    stuff: "option=argument"
- sometask: "{{ stuff }}"

Which is probably a good thing, otherwise we'll be hearing people talk about "parameter injection" vulnerabilities.

Passing complex variable values on the command line

If you have a complex playbook with lots of parameters that have to be validated it may be easier to call it from another script that will check the parameters first to ensure they make sense. Variables can be set when calling ansible-playbook with -e, e.g.:

ansible-playbook -e 'somevar=somevalue' playbook.yml

However, the --help actually states:

-e EXTRA_VARS, --extra-vars=EXTRA_VARS
                        set additional variables as key=value or YAML/JSON

so this is valid and works as expected:

ansible-playbook -e '{"somearray": [1, 2, 3], "othervar": {"objects": "too"}}' playbook.yml

JSON serialization is, of course, just an import away in most programming languages.


Cloud vs Cloud vs Physical - Benchmarking cloud & server performance with PerfKitBenchmarker

posted Jul 30, 2016, 2:27 PM by William Shallum   [ updated Jul 30, 2016, 2:51 PM ]

These days many companies provide cloud services. From the big players like Amazon (AWS), Google (GCP), and Microsoft (Azure), to the smaller ones, to the traditional hosting companies. The multitude of competing offers and pricing models can be confusing. While I don't normally deal with the financial side, I am sometimes expected to help on the technical side.

The good folks at Google along with some other companies and academic institutions have created a tool called PerfKitBenchmarker. It is written in Python and uses the cloud vendor-provided CLI tools to provision instances, run benchmarks on them, and (most importantly) terminate them afterwards[*].  Here I will show how to configure and run some simple tests using PerfKitBenchmarker (let's call it PKB from here on). I assume you use Linux or something similar. These steps are tested on CentOS 7. The installation part is mostly just me restating the documentation, but maybe you will enjoy the example configuration?

Installation

Installing PKB is mostly like installing any other python (2.7, not 3) package (except that it's not on PyPI), start with a virtualenv:

$ virtualenv-2.7 ~/pkb-ve
New python executable in /home/you/pkb-ve/bin/python
Installing Setuptools...done.
Installing Pip...done.
$ . ~/pkb-ve/bin/activate
(pkb-ve)$ pip install -U pip
Downloading/unpacking pip from https://pypi.python.org/packages/[...snip...]/pip-8.1.2.tar.gz#md5=[...snip...]
  Downloading pip-8.1.2.tar.gz (1.1MB): 1.1MB downloaded
  Running setup.py egg_info for package pip
   
    warning: no previously-included files found matching [...snip...]
    no previously-included directories found matching [...snip...]
Installing collected packages: pip
  Found existing installation: pip 1.4.1
    Uninstalling pip:
      Successfully uninstalled pip
  Running setup.py install for pip

    warning: no previously-included files found matching [...snip...]
    no previously-included directories found matching [...snip...]
    Installing pip script to /home/you/pkb-ve/bin
    Installing pip2.7 script to /home/you/pkb-ve/bin
    Installing pip2 script to /home/you/pkb-ve/bin
Successfully installed pip
Cleaning up...
(pkb-ve)$

The pip upgrade is mostly so we can use precompiled binaries (wheels) for some requirements.

Download the latest release of PKB from Github: https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/releases and untar it somewhere, for this example ~/pkb.

Now install the requirements:

(pkb-ve)$ cd ~/pkb
(pkb-ve)$ pip install -r requirements.txt
Collecting python-gflags==3.0.4 (from -r requirements.txt (line 14))
  Using cached python-gflags-3.0.4.tar.gz
[...snip...]
Successfully installed MarkupSafe-0.23 PyYAML-3.11 blinker-1.4 colorama-0.3.7 colorlog-2.6.0 futures-3.0.5 jinja2-2.8 numpy-1.11.1 pandas-0.18.1 pint-0.7.2 python-dateutil-2.5.3 python-gflags-3.0.4 pytz-2016.6.1 six-1.10.0
(pkb-ve)$

As stated above, PKB uses vendor tools. Here we will use Azure because I got some credits for free from the Visual Studio Dev Essentials Program[**].

The Azure tools are (sadly?) written in Javascript and use node.js - so you can't just reuse the virtualenv from earlier. If you don't already have node.js, packages are available from EPEL and also from SCL. In this example I am using the EPEL package (version 0.10).

If you're OK with messing up your system you can of course sudo npm install azure-cli@0.9.9 -g but let's just mess up one part of our system at a time.

(pkb-ve)$ cd ~/pkb-ve
(pkb-ve)$ npm install azure-cli@0.9.9
[...snip downloading the Internet...]
azure-cli@0.9.9 node_modules/azure-cli
[...snip dependency tree...]
(pkb-ve)$ ln -s ~/pkb-ve/node_modules/.bin/azure ~/pkb-ve/bin
(pkb-ve)$ azure
info:             _    _____   _ ___ ___
info:            /_\  |_  / | | | _ \ __|
info:      _ ___/ _ \__/ /| |_| |   / _|___ _ _
info:    (___  /_/ \_\/___|\___/|_|_\___| _____)
info:       (_______ _ _)         _ ______ _)_ _
info:              (______________ _ )   (___ _ _)
info:   
info:    Microsoft Azure: Microsoft's Cloud Platform
[...snip help...]
(pkb-ve)$

(the "azure" text in the ASCII art really is blue).

So we have installed azure-cli to a node_modules directory inside the virtualenv, and created a symlink to the CLI utility in the virtualenv's bin directory which is in the path when the virtualenv is active. PKB expects to find the CLI in the path.

Now configure the CLI to use your account. Just follow the instructions given by the Azure CLI:

(pkb-ve)$ azure account download
info:    Executing command account download
info:    Launching browser to http://go.microsoft.com/fwlink/?LinkId=254432
help:    Save the downloaded file, then execute the command
help:      account import <file>
info:    account download command OK
(pkb-ve)$ azure account import ~/Downloads/blablabla.publishsettings
info:    Executing command account import
info:    account import command OK
(pkb-ve)$

Now test if it works:

(pkb-ve)$ azure vm list
+ Getting virtual machines                                                    
info:    No VMs found
info:    vm list command OK
(pkb-ve)$

Once the account is configured, we can move on to configuring the benchmark(s).

Configuration

PKB has many benchmarks but in this case let's run a simple cross-region iperf test. PKB configuration uses YAML so create this file, let's call it iperf.yaml:

small_sea: &small_sea
  Azure:
    machine_type: Small
    zone: Southeast Asia

small_us: &small_us
  Azure:
    machine_type: Small
    zone: West US

iperf_azure: &iperf_azure
  flags:
    ip_addresses: EXTERNAL
  vm_groups:
    vm_1:
      cloud: Azure
      vm_spec: *small_us
    vm_2:
      cloud: Azure
      vm_spec: *small_sea

benchmarks:
  - iperf: *iperf_azure


The top-level keys that matter are (I think...) benchmarks and benchmark_name (e.g. iperf, ...). Other keys are ignored so they are ideal to declare anchors we can reference later. If you want to see the configuration with the references expanded try putting it into http://www.yamllint.com/.

Now we can run PKB using this file as configuration (interesting parts in bold):

(pkb-ve)$ cd ~/pkb
(pkb-ve)$ ./pkb.py --benchmark_config_file=iperf.yaml
2016-07-31 02:53:10,071 e1ddd081 MainThread INFO     Verbose logging to: /tmp/perfkitbenchmarker/runs/e1ddd081/pkb.log
2016-07-31 02:53:10,072 e1ddd081 MainThread INFO     PerfKitBenchmarker version: unknown
2016-07-31 02:53:10,072 e1ddd081 MainThread INFO     Flag values:
--benchmark_config_file=iperf.yaml
2016-07-31 02:53:10,308 e1ddd081 MainThread WARNING  The key "cloud" was not in the default config, but was in user overrides. This may indicate a typo.
2016-07-31 02:53:10,308 e1ddd081 MainThread WARNING  The key "cloud" was not in the default config, but was in user overrides. This may indicate a typo.
2016-07-31 02:53:10,309 e1ddd081 MainThread WARNING  The key "flags" was not in the default config, but was in user overrides. This may indicate a typo.
2016-07-31 02:53:10,347 e1ddd081 MainThread INFO     Running: azure -v
2016-07-31 02:53:10,600 e1ddd081 MainThread iperf(1/1) INFO     Provisioning resources for benchmark iperf
2016-07-31 02:53:10,606 e1ddd081 Thread-3 iperf(1/1) INFO     Running: azure account affinity-group create --location=Southeast Asia --label=pkbe1ddd081c5e55b7222d7 pkbe1ddd081c5e55b7222d7
[...snip...]
2016-07-31 02:54:03,702 e1ddd081 Thread-2 iperf(1/1) INFO     Ran azure network vnet create --affinity-group=pkbe1ddd0816d400dff5650 pkbe1ddd0816d400dff5650. Got return code (1).
[...snip...]
STDERR: error:   An update to your network configuration is currently underway. Please try this operation again later.
error:   network vnet create command failed
[...snip...]
2016-07-31 02:59:11,406 e1ddd081 MainThread iperf(1/1) INFO     ssh to VMs in this benchmark by name with:
  ssh -F /tmp/perfkitbenchmarker/runs/e1ddd081/ssh_config <vm_name>
  ssh -F /tmp/perfkitbenchmarker/runs/e1ddd081/ssh_config vm<index>
  ssh -F /tmp/perfkitbenchmarker/runs/e1ddd081/ssh_config <group_name>-<index>
2016-07-31 02:59:11,411 e1ddd081 MainThread iperf(1/1) INFO     Preparing benchmark iperf
[...snip...]
2016-07-31 03:05:49,850 e1ddd081 MainThread iperf(1/1) INFO     Ran ssh -A -p 22 perfkit@13.76.140.92 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /tmp/perfkitbenchmarker/runs/e1ddd081/perfkitbenchmarker_keyfile iperf --client 13.93.227.187 --port 20000 --format m --time 60 -P 1. Got return code (0).
STDOUT: ------------------------------------------------------------
Client connecting to 13.93.227.187, TCP port 20000
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[  3] local 10.32.0.4 port 40142 connected with 13.93.227.187 port 20000
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   969 MBytes   135 Mbits/sec

STDERR: Warning: Permanently added '13.76.140.92' (ECDSA) to the list of known hosts.
2016-07-31 03:05:49,855 e1ddd081 MainThread iperf(1/1) INFO     Cleaning up benchmark iperf
2016-07-31 03:05:49,855 e1ddd081 MainThread iperf(1/1) INFO     Tearing down resources for benchmark iperf
2016-07-31 03:05:49,857 e1ddd081 Thread-165 iperf(1/1) INFO     Running: azure vm delete --quiet pkb-e1ddd081-0
[...snip...]
2016-07-31 03:11:02,215 e1ddd081 MainThread INFO    
-------------------------PerfKitBenchmarker Complete Results-------------------------
[...snip...]
-------------------------PerfKitBenchmarker Results Summary-------------------------
IPERF:
  ip_type="external" receiving_machine_type="Small" runtime_in_seconds="60" sending_machine_type="Small" sending_thread_count="1"
  Throughput                          133.000000 Mbits/sec                      (receiving_zone="Southeast Asia" sending_zone="West US")
  Throughput                          135.000000 Mbits/sec                      (receiving_zone="West US" sending_zone="Southeast Asia")
  End to End Runtime                 1071.605115 seconds                      

-------------------------
For all tests: perfkitbenchmarker_version="unknown" vm_1_cloud="Azure" vm_1_image="b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_4-LTS-amd64-server-20160714-en-us-30GB" vm_1_machine_type="Small" vm_1_vm_count="1" vm_1_zone="West US" vm_2_cloud="Azure" vm_2_image="b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_4-LTS-amd64-server-20160714-en-us-30GB" vm_2_machine_type="Small" vm_2_vm_count="1" vm_2_zone="Southeast Asia"
2016-07-31 03:11:02,216 e1ddd081 MainThread INFO     Publishing 3 samples to /tmp/perfkitbenchmarker/runs/e1ddd081/perfkitbenchmarker_results.json
2016-07-31 03:11:02,216 e1ddd081 MainThread INFO     Benchmark run statuses:
------------------------
Name   UID     Status  
------------------------
iperf  iperf0  SUCCEEDED
------------------------
Success rate: 100.00% (1/1)
2016-07-31 03:11:02,217 e1ddd081 MainThread INFO     Complete logs can be found at: /tmp/perfkitbenchmarker/runs/e1ddd081/pkb.log
(pkb-ve)$


Just ignore the scary WARNING parts. The cloud and the flags are usually specified on the command line (the same vm_spec can have configurations for multiple clouds). I'm not sure I like troubleshooting the cascading flags so I prefer to just configure everything in the config file[***].

Since operations in "the Cloud" take some time to respond, some errors are possible (e.g. see above). PKB will retry the operations after a slight delay.

As you can see, it takes a long time to finish, but at least you didn't have to click on anything and it cleans up after itself.

Configuration - Static VM

Contrary to popular belief, not everyone has moved to "the Cloud" yet. You may have some machines in your own datacenter or on some hosting provider that is insufficiently "cloudy" and is unsupported by PKB. Fear not! You can run PKB on those machines by using the static_vms feature ("static" probably means PKB will not provision them?). Here we will run the same iperf benchmark as above. This will connect to a VPS and run iperf between it and an Azure VM. Let's call this file iperf-static.yaml:

small_sea: &small_sea
  Azure:
    machine_type: Small
    zone: Southeast Asia

small_us: &small_us
  Azure:
    machine_type: Small
    zone: West US

static_vms:
  us-vps: &us-vps
    ip_address: your.ip.here
    user_name: your_user
    ssh_private_key: /path/to/ssh/key/here # hopefully passwordless?
    ssh_port: 22
    os_type: debian # It's actually Alpine Linux, not Debian, but...
    install_packages: False # I installed iperf and configured the firewall before running PKB.

iperf_azure_vps: &iperf_azure_vps
  flags:
    ip_addresses: EXTERNAL
  vm_groups:
    vm_1:
      cloud: Azure
      vm_spec: *small_us
    vm_2:
      static_vms:
        - *us-vps

benchmarks:
  - iperf: *iperf_azure_vps

Run it like before (relevant parts and commands run on static VM highlighted):

$ ./pkb.py --benchmark_config_file=iperf-static.yaml
2016-07-31 03:46:31,497 558636a2 MainThread INFO     Verbose logging to: /tmp/perfkitbenchmarker/runs/558636a2/pkb.log
2016-07-31 03:46:31,497 558636a2 MainThread INFO     PerfKitBenchmarker version: unknown
2016-07-31 03:46:31,497 558636a2 MainThread INFO     Flag values:
--benchmark_config_file=iperf-static.yaml
2016-07-31 03:46:31,667 558636a2 MainThread WARNING  The key "static_vms" was not in the default config, but was in user overrides. This may indicate a typo.
2016-07-31 03:46:31,668 558636a2 MainThread WARNING  The key "cloud" was not in the default config, but was in user overrides. This may indicate a typo.
2016-07-31 03:46:31,668 558636a2 MainThread WARNING  The key "flags" was not in the default config, but was in user overrides. This may indicate a typo.
2016-07-31 03:46:31,708 558636a2 MainThread INFO     Running: azure -v
2016-07-31 03:46:31,925 558636a2 MainThread iperf(1/1) INFO     Provisioning resources for benchmark iperf
[...snip...]
2016-07-31 03:47:59,046 558636a2 Thread-9 iperf(1/1) INFO     VM: your.ip.here
2016-07-31 03:47:59,048 558636a2 Thread-9 iperf(1/1) INFO     Running: ssh -A -p 22 your_user@your.ip.here -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /path/to/ssh/key/here hostname
2016-07-31 03:48:01,412 558636a2 Thread-9 iperf(1/1) INFO     Running: ssh -A -p 22 your_user@your.ip.here -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /path/to/ssh/key/here mkdir -p /tmp/pkb
[...snip...]
2016-07-31 03:51:25,663 558636a2 MainThread iperf(1/1) INFO     Preparing benchmark iperf
[...snip...]
2016-07-31 03:54:54,948 558636a2 MainThread iperf(1/1) INFO     Running: ssh -A -p 22 perfkit@137.135.47.113 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /tmp/perfkitbenchmarker/runs/558636a2/perfkitbenchmarker_keyfile nohup iperf --server --port 20000 &> /dev/null& echo $!
2016-07-31 03:54:58,539 558636a2 MainThread iperf(1/1) INFO     Running: ssh -A -p 22 your_user@your.ip.here -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /path/to/ssh/key/here nohup iperf --server --port 20000 &> /dev/null& echo $!
2016-07-31 03:55:00,982 558636a2 MainThread iperf(1/1) INFO     Running benchmark iperf
2016-07-31 03:55:00,985 558636a2 MainThread iperf(1/1) INFO     Iperf Results:
2016-07-31 03:55:00,986 558636a2 MainThread iperf(1/1) INFO     Running: ssh -A -p 22 perfkit@137.135.47.113 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /tmp/perfkitbenchmarker/runs/558636a2/perfkitbenchmarker_keyfile iperf --client your.ip.here --port 20000 --format m --time 60 -P 1
2016-07-31 03:56:05,141 558636a2 MainThread iperf(1/1) INFO     Ran ssh -A -p 22 perfkit@137.135.47.113 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /tmp/perfkitbenchmarker/runs/558636a2/perfkitbenchmarker_keyfile iperf --client your.ip.here --port 20000 --format m --time 60 -P 1. Got return code (0).
STDOUT: ------------------------------------------------------------
Client connecting to your.ip.here, TCP port 20000
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[  3] local 10.32.0.4 port 42062 connected with your.ip.here port 20000
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  3576 MBytes   500 Mbits/sec

STDERR: Warning: Permanently added '137.135.47.113' (ECDSA) to the list of known hosts.

2016-07-31 03:56:05,144 558636a2 MainThread iperf(1/1) INFO     Running: ssh -A -p 22 your_user@your.ip.here -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /path/to/ssh/key/here iperf --client 137.135.47.113 --port 20000 --format m --time 60 -P 1
2016-07-31 03:57:18,261 558636a2 MainThread iperf(1/1) INFO     Ran ssh -A -p 22 your_user@your.ip.here -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /path/to/ssh/key/here iperf --client 137.135.47.113 --port 20000 --format m --time 60 -P 1. Got return code (0).
STDOUT: ------------------------------------------------------------
Client connecting to 137.135.47.113, TCP port 20000
TCP window size: 0.04 MByte (default)
------------------------------------------------------------
[  3] local your.ip.here port 51766 connected with 137.135.47.113 port 20000
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  7949 MBytes  1111 Mbits/sec

STDERR: Warning: Permanently added 'your.ip.here' (ECDSA) to the list of known hosts.
2016-07-31 03:57:18,266 558636a2 MainThread iperf(1/1) INFO     Cleaning up benchmark iperf
2016-07-31 03:57:18,266 558636a2 MainThread iperf(1/1) INFO     Running: ssh -A -p 22 perfkit@137.135.47.113 -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /tmp/perfkitbenchmarker/runs/558636a2/perfkitbenchmarker_keyfile kill -9 2361
2016-07-31 03:57:22,211 558636a2 MainThread iperf(1/1) INFO     Running: ssh -A -p 22 your_user@your.ip.here -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /path/to/ssh/key/here kill -9 2472
2016-07-31 03:57:28,819 558636a2 MainThread iperf(1/1) INFO     Tearing down resources for benchmark iperf
[...snip...]
2016-07-31 04:01:11,247 558636a2 MainThread INFO    
-------------------------PerfKitBenchmarker Complete Results-------------------------
[...snip...]
-------------------------PerfKitBenchmarker Results Summary-------------------------
IPERF:
  ip_type="external" runtime_in_seconds="60" sending_thread_count="1"
  Throughput                          500.000000 Mbits/sec                      (receiving_machine_type="None" receiving_zone="Static - your_user@your.ip.here" sending_machine_type="Small" sending_zone="West US")
  Throughput                         1111.000000 Mbits/sec                      (receiving_machine_type="Small" receiving_zone="West US" sending_machine_type="None" sending_zone="Static - your_user@your.ip.here")
  End to End Runtime                  879.311017 seconds                      

-------------------------
For all tests: perfkitbenchmarker_version="unknown" vm_1_cloud="Azure" vm_1_image="b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_4-LTS-amd64-server-20160714-en-us-30GB" vm_1_machine_type="Small" vm_1_vm_count="1" vm_1_zone="West US" vm_2_cloud="Static" vm_2_image="None" vm_2_vm_count="1" vm_2_zone="Static - your_user@your.ip.here"
2016-07-31 04:01:11,249 558636a2 MainThread INFO     Publishing 3 samples to /tmp/perfkitbenchmarker/runs/558636a2/perfkitbenchmarker_results.json
2016-07-31 04:01:11,250 558636a2 MainThread INFO     Benchmark run statuses:
------------------------
Name   UID     Status  
------------------------
iperf  iperf0  SUCCEEDED
------------------------
Success rate: 100.00% (1/1)
2016-07-31 04:01:11,251 558636a2 MainThread INFO     Complete logs can be found at: /tmp/perfkitbenchmarker/runs/558636a2/pkb.log
(pkb-ve)$

The process is similar except that the static VM is neither provisioned nor cleaned up. If install_packages is not False (and maybe even if it is true?), the user must have passwordless sudo access. Fortunately the iperf benchmark does not need any privileged commands.

Conclusion

If your office network is still using Fast Ethernet, it has less bandwidth than a link between West US and Southeast Asia on the Internet.

For more information e.g. about configuration for the individual benchmarks, the actions each benchmark takes, please look at the source code.

References



[*] Terminating them is important if you like your money and wish to keep it.
[**] Which is pretty cool. I knew paying for Windows 7 Pro would pay off one day.
[***] From the documentation (emphasis added):

  --zones: A list of zones within which to run PerfKitBenchmarker. This is
    specific to the cloud provider you are running on. If multiple zones are
    given, PerfKitBenchmarker will create 1 VM in zone, until enough VMs are
    created as specified in each benchmark. The order in which this flag is
    applied to VMs is undefined.




Generating Certificate Requests (CSRs) on Windows

posted Apr 9, 2016, 10:39 PM by William Shallum   [ updated Apr 9, 2016, 10:41 PM ]

I use Windows. If I want to generate a CSR using OpenSSL it is easy and there are lots of guides on the Internet you can copy commands from. In fact I have done it multiple times and can probably do it from memory. But what if I want to generate a CSR using the Windows GUI tools?

Windows has decent support for using a "Certificate Enrollment Policy Server", but as this is going to be used by me to authenticate to my (Linux) VPS, that is not an option here.

Steps:
  • Run certmgr.msc
  • Right click on the Personal > Certificates folder, select All Tasks > Advanced Operations > Create Custom Request...
  • When prompted for a Certificate Enrollment Policy, select "Proceed without enrollment policy"
  • Select "No Template (CNG Key)" for key and PKCS#10 for the request format.
  • Click on the Details button then Properties to customize the requested certificate attributes like Common Name, Key Usage, Key Size, etc.
    • When specifying subject DN, the top entry in the list box is actually the last (most specific) DN element which is usually the Common Name. This also goes for the "Full DN" text box where the most specific part is specified first.
    • Remember when used for client authentication to strongSwan, always specify a DNS subjectAltName that matches the CN since that is required by the EAP-TLS validator in strongSwan.
  • Click next and save the request somewhere.

Disabling gnome-keyring-daemon SSH agent on MATE Desktop

posted Feb 26, 2016, 5:29 AM by William Shallum   [ updated Feb 26, 2016, 5:33 AM ]

The mate-session-manager starts gnome-keyring-daemon by default with all components enabled, including the SSH agent. This is less than optimal since the SSH agent lacks a lot of features compared to OpenSSH's ssh-agent (e.g. support for ECDSA and Ed25519 keys).

To stop this from happening you need to change the value of a key in gsettings:

$ gsettings get org.mate.session gnome-compat-startup
['smproxy', 'keyring']
$ gsettings set org.mate.session gnome-compat-startup "['smproxy']"
$ gsettings get org.mate.session gnome-compat-startup
['smproxy']

That's not all. MATE actually includes autostart entries - enabled by default - for all 3 components of gnome-keyring-daemon (which makes it a little curious why mate-session-manager starts it unconditionally). It just takes a few clicks to disable:

$ mate-session-properties
# uncheck SSH Key Agent and maybe others.



Android One Rdio download to SD card

posted Aug 21, 2015, 10:46 AM by William Shallum   [ updated Aug 21, 2015, 10:47 AM ]

Since my Android One phone has limited external memory, I thought buying a 16GB MicroSD card would help. I was wrong.

Having not used Android since the Gingerbread days, the MicroSD seems a lot cleaner now after the changes. A. LOT. CLEANER. Which is a good thing, until something you want to put there doesn't get put there*, and that was the case with the Rdio cache. The app claims that it wants to put it in external storage, but that doesn't seem to work. Fortunately, it allows the user to explicitly specify a location (see previous link).

To use this first, find out where the real external storage is mounted. On my phone it was on /storage/sdcard1 (and not e.g. /sdcard or /storage/sdcard0, and obviously not /storage/emulated/legacy - no wonder the app got confused). The example on the screen (/sdcard) would have you believe that what you put there should be the root path of the storage, but if you tried that, you would get errors on every download.

After looking at logcat output to determine where exactly it is trying to create the data, the proper directory to put in the custom storage location is the Android/data directory inside your external storage, so in my case that would be /storage/sdcard1/Android/data.

Cheap phones: for people whose time isn't worth much**.

[*]: e.g. games with hundreds of megs in assets, but that can't be helped.
[**]: saying "just root it" only proves the point.

32 bit golang Trace/Breakpoint trap: modify_ldt ENOSYS

posted Aug 4, 2015, 9:04 AM by William Shallum   [ updated Aug 4, 2015, 9:06 AM ]

If you're like me: a bit too cheap to get a large VPS and too miserly to waste limited RAM by using a 64 bit OS you may run into something like this:

vm:~$ gdrive
Trace/breakpoint trap


Where gdrive is https://github.com/prasmussen/gdrive - it is written in the Go programming language, which would explain the rather interesting behavior below.

It worked before, so something must be wrong. A short strace later showed:

vm:~$ strace /usr/local/bin/gdrive
execve("/usr/local/bin/gdrive", ["/usr/local/bin/gdrive"], [/* 11 vars */]) = 0
modify_ldt(1, {entry_number:7, base_addr:0x84d8548, limit:1048575, seg_32bit:1,
contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}, 1
6) = -1 ENOSYS (Function not implemented)
--- SIGTRAP {si_signo=SIGTRAP, si_code=SI_KERNEL} ---
+++ killed by SIGTRAP +++
Trace/breakpoint trap


Some searching around led to this LKML discussion: https://lkml.org/lkml/2015/7/23/243

It seems on this kernel:

vm:~$ uname -a
Linux vm 3.18.19-0-virtgrsec #1-Alpine SMP Fri Jul 31 11:09:05 GMT 2015 i686 Linux


The patch mentioned was applied. Luckily the syscall can still be reenabled by

# echo 1 > /proc/sys/kernel/modify_ldt


After re-enabling the syscall, the program works fine.


Docker on LVM Thin Pool

posted Jul 25, 2015, 12:23 AM by William Shallum   [ updated Jul 25, 2015, 12:24 AM ]

These are notes on how to run docker on top of an LVM thin pool, which is nice if your system already has LVM and you have some spare space in the VG.

Tested on Arch. Instruction removes all docker data so maybe don't do it if you have data you care about?
  1. Create the thin pool from two LVs: one (larger) for data, the other (smaller) for metadata. man 7 lvmthin has examples.
  2. Customize the docker.service systemd unit file to add storage driver args specifying that it should use the thin pool.
    1. systemctl stop docker
    2. rm -rf /var/lib/docker you don't have anything there you want to keep do you?
    3. systemctl edit --full docker.service (much better than the copy / daemon-reload dance).
    4. Add to the ExecStart line: -s devicemapper --storage-opt dm.thinpooldev=name-under-dev-mapper and by name-under-dev-mapper I mean look under your /dev/mapper directory and use the name corresponding to your thin LV (will look like vgname-lvname). Using /dev/vgname/lvname leads to an interesting error.
    5. systemctl start docker
  3. journalctl -e any errors?
  4. docker info

Enjoy:

Storage Driver: devicemapper
 Pool Name: vg_laptop-docker--thinpool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 21.76 MB
 Data Space Total: 21.47 GB
 Data Space Available: 21.45 GB
 Metadata Space Used: 266.2 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Library Version: 1.02.100 (2015-06-30)

References

  1. https://docs.docker.com/reference/commandline/daemon/#storage-driver-options
  2. lvmthin manpage
  3. http://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/

UEFI Linux Install on Acer Aspire ES1-111M

posted Jul 2, 2015, 9:44 AM by William Shallum   [ updated Jul 5, 2015, 1:16 AM ]

Some notes on installing Linux (in Insecure[*] UEFI boot mode) on this Acer netbook. If you are trying to do this, you should know what you're doing, so these are just notes, not instructions.
  • Secure boot settings can only be changed when a BIOS Supervisor password is set. This does not seem to require actually saving the Supervisor password, so just setting it, modifying secure boot settings, then clearing it again should work.
  • If getting into BIOS is hard, Windows can be set to reboot to BIOS.
  • Keys for BIOS access are F2 for Setup and F12 for boot menu (can be enabled in BIOS).
  • Even with Secure Boot disabled, adding a boot entry using efibootmgr will not "take" if the .efi file is not trusted by Secure Boot (!). Adding a boot entry with an untrusted loader will result in it simply being ignored and the system still booting into Windows.
  • In order to trust the loader, Secure Boot must be enabled.
  • So AFTER installing Linux and setting up the loader in the ESP, the steps are:
    • Get into BIOS
    • Set Supervisor password
    • Ensure Secure Boot is on
    • Trust the loader you added to the ESP ("Select an UEFI file as trusted for execution")
    • Turn off Secure Boot
    • (Optionally) clear Supervisor password
    • Save BIOS settings and restart
    • Boot to Linux again (e.g. via external drive with EFI\Boot\bootx64.efi or an EFI bootable CD) and register the trusted loader as a boot option using efibootmgr
    • NOTE: steps written from memory, not actually tested. It could be that adding the boot option needs to be done under Secure Boot. In that case (if you use rEFInd), also add the kernel as trusted (may need to be copied to ESP and renamed to have a .efi extension).
  • rEFInd is Secure Boot compliant and will not load an untrusted kernel when Secure Boot is enabled.
  • The hard drive ("WD5000LPVX" - WD Scorpio Blue) idle timer interacts rather poorly with Linux. Try http://idle3-tools.sourceforge.net/ (AT YOUR OWN RISK).

[*] Secure boot should also be possible, but when using a distribution that does not sign their kernels that seems to be more trouble than it's worth.


xrandr Set Default Scaling Mode

posted Apr 18, 2015, 3:10 PM by William Shallum   [ updated Jul 5, 2015, 1:17 AM ]

Among other annoyances, the TV monitor does not react well (displays a fringing effect between high contrast areas) when the xrandr scaling mode is set to Full (using the nouveau driver here). Sadly this is the default. Even sadder, there is no place in the xorg.conf to actually fix this (xrandr being a dynamic mechanism after all, to support adding displays dynamically), so the fix is to run xrandr on every X startup.

The best place to do that is in GDM (version 3) initialization since that will also fix the GDM login display.

On my CentOS 7 system this is /etc/gdm/Init/Default (but don't edit that). The docs say that GDM will load the Default script if there is no display-specific script. Let's just add a display-specific script /etc/gdm/Init/:0:

#!/bin/sh

XRANDR=/usr/bin/xrandr
if [ -x "$XRANDR" ] ; then
  if "$XRANDR" | grep '^DVI-I-1 connected' >/dev/null 2>&1 ; then
    "$XRANDR" --output DVI-I-1 --set "scaling mode" "None"
  fi
fi

exec /etc/gdm/Init/Default


This was not a problem when CentOS 6 was installed on the machine so I can only guess there were some interesting changes in the intervening versions.

References:

1-10 of 63