Creating a persistent ssh tunnel in Ubuntu

In situations when no VPN is either not available or you just think it’s an overkill to configure an ssh tunnel can be a pretty good alternative. An SSH tunnel works by setting up an ssh connetion between two hosts and use that connection to transport normal network traffic. On one side of the tunnel, the OpenSSH software takes anything that is sent to a specific port and sends it over to the other side of the connection. When the packets arrive on the target side, the OpenSSH software forwards them to the correct local port. Natrually the traffic is encrypted on it’s way, thereby creating a miniature VPN.

Setting up an ssh tunnel is quite straigt forward, for example at the terminal:

ssh -NL 8080:127.0.0.1:80 root@192.168.10.10

Let’s break that down:

There’s really only one problem with this. Whenever the ssh connection breaks the tunnel will break and you have to rerun that command again. Depending on your situation this might be a problem and that’s why autossh exists.

Autossh

Autossh is a tool that sets up a tunnel and then checks on it every 10 seconds. If the tunnel stopped working autossh will simply restart it again. So instead of running the command above you could run:

autossh -NL 8080:127.0.0.1:80 root@192.168.10.10

Note: starting an ssh tunnel with autossh assumes that you have set up public key authentication between the client and server since there’s no way for autossh to ask you for the password. Especially if you want to follow the rest of this article and have the tunnels start automatically. Please read more here  for details on how to setup public key authentication.

Start automatically

Having autossh monitor your tunnels is a great advantage, but it still requires a command to be run whenever autossh itself would die, such as after a reboot. If you want your ssh tunnels to persist over a reboot you would need to create a startup script. Since Ubuntu 9.10 (Karmic) Ubuntu have used Upstart as its main mechanism to handle startup scripts so the rest of this article will discuss how such a script can be created.

I wanted to achieve three things with my startup script. First, I’d like my tunnel to come up when the computer boots. Second, I’d like to be able to keep configuration for multiple tunnels in one file. And lastly, I’d like to be able to control (start/stop) my tunnels without having to kill individual ssh processes.
Three files are created:

First, /etc/init/autossh.conf:

description "autossh tunnel"
author "Erik Torsner "

start on (local-filesystems and net-device-up IFACE=eth1 and net-device-up IFACE=wlan1)
stop on runlevel [016]

pre-start script
NUMHOSTS=$(egrep -v '^[[:space:]]*$' /etc/autossh.hosts | wc -l)
for i in `seq 1 $NUMHOSTS`
do
start autossh_host N=$i
done
end script

Second, /etc/init/autossh_host.conf

stop on stopping autossh

respawn

instance $N
export HOST=$N

script
ARGS=$(head -$N /etc/autossh.hosts | tail -1)
exec autossh $ARGS
end script

And lastly, the config file /etc/autossh.hosts

-NL 8080:127.0.0.1:80 root@host1.example.com
-NL 8080:127.0.0.1:80 root@host2.example.com

Rationale for each file

/etc/init/autossh.conf
This is the main startup script. The first parts of the file sets the conditions that triggers this script to start or stop. I’ve told my script to start as soon as the local filesystem is ready and when mu network interfaces are up. Chances are that you want to modify this to suit your computer, at least the name of the network interfaces.
The actual script part is a simple bash script that checks the contents of a config file to determine how many tunnels that needs to be started. It then starts a numer of autossh_host jobs with an individual id that happens to correspond a the line number. It would have been nice to just start the autossh processes directly from this script, but autossh is a blocking command and only the first line would be executed on startup. Dropping the process to the background using a & character at the end would work but the resulting autossh processes would be out of our controll.

/etc/init/autossh_host.conf
This is the job where the interesting part actually happens. Note that this script doesn’t have any “start on” stanza at all, it’s only ever started when the main job tells it to start. The interesting part is its “stop on” stanza that simply says that it should stop whenever the parent job is stopped. Using this mechanism all child jobs can be controlled via the parent job.

The script part uses a very naive method of picking out the correct line from /etc/autossh.hosts and use that line as the argument for autossh. Note that with this approach, the config file /etc/autossh.hosts can’t contain any empty lines or comments at all. Room for improvement.

/etc/autossh.hosts
This is just a config file with one row per ssh tunnel to create. Each line consists of the entire parameter line sent to autossh. Again, please note that the parsing of this file is very naive and won’t allow for any blank lines or comments at all.
Starting the tunnels
If everything is correctly set up, you should now be able to start your tunnels using:

sudo start autossh

and stopping them with

sudo stop autossh

If you have problems, check out the logfiles under /var/log/upstart. There should be one logfile for each script/instance, so expect to find autossh.log as well as many autossh_host-N.log.

Questions? Did I miss something? Is my solution entierly wrong? Let me know in the comments.

20 comments

  1. Thanks, this was helpful.

    When setting this up I got caught out by the fact that the destination host wasn’t in the root user’s known_hosts file. This caused things to fail silently (didn’t produce any output in the error logs), so it took me a while to figure out what was going on. I had tested the autossh command manually – but not as the root user, since I generally avoid running commands as root unless absolutely necessary.

    TLDR; Test all connections manually as the *root* user first to ensure known_hosts gets populated.

  2. Hi !

    It doesn’t work for me.
    I did exactely what you wrote. I only changed the names of my interfaces (eth0 and wlan0). I rebooted but autossh wans’t running.
    I even started it with : sudo start autossh
    The tunel wasn’t on.

    here are the rights to my files :
    -rw-r–r– 1 root root 317 nov. 1 23:32 autossh.conf
    -rw-r–r– 1 root root 146 nov. 1 23:33 autossh_host.conf

    Any clue ?

    1. If the public key authentication between the client and server was created by using another user than root; say myuser, then you need to change the line that says:
      exec autossh $ARGS
      of the file /etc/init/autossh_host.conf to
      exec su myuser -c “autossh $ARGS”

      1. That’s good, but what if myuser’s public key auth is protected by a password ? It doesn’t work and my log clearly states “Permission denied”. How can I bypass it for this command only ?

    1. Hi John,

      So far I haven’t had to deal with systemd for any of my projects. As far as I know the Upstart scripts are still working even on the newest Ubuntu (might be wrong). If (or when) I get to the point where I have to learn it, I’ll write another post about it and link it from here. But right now, You’re probably already know more than I do.

      Sorry I can’t help.

      1. I am geting the below error. I am trying to run this using a different user. config

        exec su hp-admin -c “autossh $ARGS” (this per your advice above to use another user and key)

        actual auto ssh command
        -NL 0.0.0.0:2222:localhost:65534 hp-admin@10.10.2.210 -i /home/hp-admin/.ssh/sshkey

        Feb 8 14:51:01 ubuntu kernel: [ 5557.059286] init: autossh_host (1) main process (1794) terminated with status 2
        Feb 8 14:51:01 ubuntu kernel: [ 5557.059296] init: autossh_host (1) main process ended, respawning
        Feb 8 14:51:01 ubuntu kernel: [ 5557.066592] init: autossh_host (1) main process (1799) terminated with status 2
        Feb 8 14:51:01 ubuntu kernel: [ 5557.066602] init: autossh_host (1) respawning too fast, stopped
        Feb 8 14:53:40 ubuntu autossh[1806]: starting ssh (count 1)
        Feb 8 14:53:40 ubuntu autossh[1806]: ssh child pid is 1809
        Feb 8 14:56:06 ubuntu autossh[1806]: ssh exited with status 129; autossh exiting

        If i just run the below while logged in as hp-admin it works….this is a default Ubuntu 14 install so root is not used:
        autossh -M 5122 -N -R 0.0.0.0:2222:localhost:65534 hp-admin@10.10.2.210 -i /home/hp-admin/.ssh/sshkey

  3. I am getting these errors. Also I am wanting to run it with a user other than root and the key is stored under that users home/…./.ssh/ directory

    Here is the full CLI command i can run to get it to work outside this script:

    autossh -M 5122 -N -R 0.0.0.0:2222:localhost:65534 hp-admin@10.10.2.210 -i ~/.ssh/sshkey

    Feb 8 14:51:01 ubuntu kernel: [ 5557.059296] init: autossh_host (1) main process ended, respawning
    Feb 8 14:51:01 ubuntu kernel: [ 5557.066592] init: autossh_host (1) main process (1799) terminated with status 2
    Feb 8 14:51:01 ubuntu kernel: [ 5557.066602] init: autossh_host (1) respawning too fast, stopped

  4. I am getting this error when trying to run the scripts. I am also using a non-root account and key. works fine if i just copy and paste the autossh command

    Feb 9 08:45:45 ubuntu kernel: [70050.122543] init: autossh_host (1) main process (3172) terminated with status 2
    Feb 9 08:45:45 ubuntu kernel: [70050.122556] init: autossh_host (1) respawning too fast, stopped

  5. Thanks a lot, this is very useful article

    But having a single file for configuration is not always suitable, following modifications will allow you to have single file OR a directory of configuration files

    change
    NUMHOSTS=$(egrep -v ‘^[[:space:]]*$’ /etc/autossh.hosts | wc -l)
    TO
    NUMHOSTS=$(find /etc/autossh.hosts -exec egrep -v ‘^[[:space:]]*$’ “{}” | wc -l)

    AND

    ARGS=$(head -$N /etc/autossh.hosts | tail -1)
    TO
    ARGS=$(find /etc/autossh.hosts | xargs cat 2>/dev/null | head -$N | tail -1)

  6. Works well on my new Ubuntu 13.10 install. Ubuntu seems to have moved on to other auto start mechanisms, but I was stuck with 13.10 for hardware reasons and your solution was a perfect fit. Many thanks.

    1. Yeah, I probably need to update this article to reflect recent Ubuntu versions. Thanks for reminding me and great to hear it works out on your system! I really appreciate the feedback.

Leave a Reply to erik@torgesta.com Cancel reply

Your email address will not be published. Required fields are marked *