SSH ProxyJump: Reach a Server Through a Jump Host

Some servers accept SSH from one address only. A firewall in front of them allows port 22 from the office, or from one server with a static IP, and drops everything else. From home or from a hotel, ssh just waits and then gives up. Many people fix this by copying their private key to the server with the static IP and logging in from there. It works, but now the private key lives on a second machine.

SSH has a better tool for this job: ProxyJump. Your laptop connects through the server with the static IP, and the key never leaves the laptop. This post shows the setup, the errors you will meet and how to limit what the jump host can do.

I tested everything with three Alpine Linux containers and OpenSSH 10.3. There is a laptop, a jump host called jump (198.51.100.10, SSH on port 222) and a server called web1 (198.51.100.20). A firewall on web1 accepts port 22 only from the jump host. All output below is real. ProxyJump needs OpenSSH 7.3 or newer on the laptop.

The problem

This is a direct login from the laptop to web1:

$ ssh -o ConnectTimeout=10 deploy@198.51.100.20 hostname
ssh: connect to host 198.51.100.20 port 22: Operation timed out

The firewall drops the packets without an answer, so ssh waits until it times out. I added ConnectTimeout=10 to keep the wait short. On Linux systems with glibc, the message says "Connection timed out".

Which key goes where

With ProxyJump, your laptop logs in to both servers itself. So both servers need your public key, and only the laptop has the private key:

One command: ssh -J

The -J option names the jump host. Its port goes after a colon:

$ ssh -J deploy@198.51.100.10:222 deploy@198.51.100.20 hostname
web1

Your laptop logs in to jump and asks it to open a TCP connection to port 22 on web1. Then the laptop starts a second SSH session through that connection. This second session is encrypted from the laptop to web1. The jump host passes the bytes on, but it cannot read them, and it never sees your key.

If web1 still accepts passwords, you can install your public key through the jump host as well. ssh-copy-id passes -o options on to ssh:

ssh-copy-id -o ProxyJump=deploy@198.51.100.10:222 deploy@198.51.100.20

It asks for the password of deploy on web1 once and ends with Number of key(s) added: 1. If web1 accepts keys only, somebody who can already log in has to add the line for you.

Put it in ~/.ssh/config

The -J form is long to type every time. Put both servers in ~/.ssh/config instead:

Host jump
    HostName 198.51.100.10
    Port 222
    User deploy
Host web1
    HostName 198.51.100.20
    User deploy
    ProxyJump jump

ProxyJump jump points to the Host jump entry, so the port and user of the jump host come from there. Add an IdentityFile line to both entries if your key does not have a default name. Now the short name is enough:

$ ssh web1 'hostname; echo "$SSH_CLIENT"'
web1
198.51.100.10 55624 22

SSH_CLIENT shows where the connection came from, as web1 sees it. That is the address of the jump host, not the laptop. This is why the firewall on web1 lets the connection in. The jump host also has no copy of the private key:

$ ssh jump ls -a .ssh
.
..
authorized_keys

Every tool that runs ssh reads the same config file. So scp and rsync go through the jump host without any extra option:

$ scp site/index.html web1:/tmp/
$ rsync -ai site/ web1:site/
created directory site
cd+++++++++ ./
<f+++++++++ index.html
<f+++++++++ style.css

Without a config entry, scp accepts -J too, and rsync takes it in -e 'ssh -J deploy@198.51.100.10:222'.

The jump host also resolves the name of the target. In my lab, only jump knows the name web1.internal, from its /etc/hosts. The laptop cannot resolve it, but the login still works. This helps with private DNS names inside a data centre:

$ getent hosts web1.internal
$ ssh -J jump deploy@web1.internal hostname
web1

What ssh -v shows

When something goes wrong, ssh -v shows both connections. These are the important lines out of 178:

$ ssh -v web1 true
debug1: Setting implicit ProxyCommand from ProxyJump: ssh -v -W '[%h]:%p' jump
debug1: Executing proxy command: exec ssh -v -W '[198.51.100.20]:22' jump
debug1: Connecting to 198.51.100.10 [198.51.100.10] port 222.
Authenticated to 198.51.100.10 ([198.51.100.10]:222) using "publickey".
Authenticated to 198.51.100.20 (via proxy) using "publickey".

ProxyJump is a short form of ProxyCommand. Your ssh starts a second ssh to the jump host with -W host:port. That option connects the input and output of the second ssh to the target. The first login is to the jump host on port 222, and the second one is to web1 "via proxy".

Because there are two ssh processes, a key with a passphrase is unlocked twice. In my test, ssh web1 asked for the passphrase two times. After ssh-add, the agent answered both logins and ssh did not ask at all.

OpenSSH older than 7.3 has no ProxyJump, but it has -W. You can write the same thing by hand:

$ ssh -o "ProxyCommand=ssh -p 222 -W %h:%p deploy@198.51.100.10" deploy@198.51.100.20 hostname
web1

For more than one hop, give -J or ProxyJump a list of hosts separated by commas. SSH connects to them in the order of the list.

When it does not work

Wrong port on the jump host

I made this mistake on a real server. The jump host listened on port 222, but my config did not say so:

$ ssh web1 hostname
ssh: connect to host 198.51.100.10 port 22: Connection refused
Connection closed by UNKNOWN port 65535

The last line comes from the outer ssh. It only says that the proxy failed, so always read the line above it. Here nothing listens on port 22, so the jump host refuses the connection. When a firewall drops the packets instead, you get a timeout, and ssh -v stops at the Connecting to line. In both cases, check the Port line of the jump host. On the server, ss -tlnp shows the port that sshd listens on.

Wrong file permissions

SSH checks file permissions on both sides. On the laptop, it refuses a config file that other users can change:

$ chmod 664 ~/.ssh/config
$ ssh web1 hostname
Bad owner or permissions on /home/awie/.ssh/config

It also ignores a private key that other users can read. (BatchMode=yes stops ssh from asking for a password instead.)

$ chmod 644 ~/.ssh/id_ed25519
$ ssh -o BatchMode=yes jump hostname
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/home/awie/.ssh/id_ed25519' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/home/awie/.ssh/id_ed25519": bad permissions
deploy@198.51.100.10: Permission denied (publickey,password,keyboard-interactive).

On the server, sshd ignores authorized_keys when other users can write to it, to ~/.ssh or to the home directory. This case is harder to find, because the client only says "Permission denied". The reason is in the server log (/var/log/secure, /var/log/auth.log or journalctl, depending on the distribution):

Authentication refused: bad ownership or modes for file /home/deploy/.ssh/authorized_keys

On systems where the umask is 002, every new file and directory is writable by the group. So a ~/.ssh or authorized_keys that you create by hand fails this check. ssh-copy-id creates both with safe permissions, but it does not fix files that already exist. In my test, it added the key to a group-writable authorized_keys, and the login still failed. These commands set the right permissions:

# on the laptop
chmod 700 ~/.ssh
chmod 600 ~/.ssh/config ~/.ssh/id_ed25519
# on each server, as the user you log in as
chmod go-w ~
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

With ProxyJump, the server part applies to the jump host and to the target.

Forwarding is turned off on the jump host

$ ssh web1 hostname
channel 0: open failed: administratively prohibited: open failed
stdio forwarding failed
Connection closed by UNKNOWN port 65535

The login to the jump host worked, but its sshd refused to open the connection to web1. Check AllowTcpForwarding in /etc/ssh/sshd_config on the jump host. The OpenSSH default is yes, but the config file that Alpine Linux ships sets it to no. A new Alpine jump host gives exactly this error.

Limit what the jump host can reach

A jump host that forwards to any address gives every user on it a way into your whole internal network. Two lines in its /etc/ssh/sshd_config limit that:

AllowTcpForwarding local
PermitOpen 198.51.100.20:22

local allows the forwarding that -W uses and refuses remote forwarding (ssh -R). PermitOpen lists the only targets that the jump host connects to. Put both lines above any Match block, test the file with sshd -t and reload sshd. Now web1 still works, and the rest fails:

$ ssh web1 hostname
web1
$ ssh -J jump deploy@198.51.100.5 hostname
channel 0: open failed: administratively prohibited: open failed
stdio forwarding failed
Connection closed by UNKNOWN port 65535
$ ssh -o ExitOnForwardFailure=yes -R 8080:localhost:80 jump true
Error: remote port forwarding failed for listen port 8080

PermitOpen compares the host exactly as the client sends it and does not resolve names. With only the address in the list, ssh -J jump deploy@web1.internal failed with the same "administratively prohibited" error. If you connect by name, list the name too.

Why not agent forwarding or a copied key

A private key that you copy to the jump host stays there. Anyone who gets root on that server can take it and use it, even years later.

ssh -A (agent forwarding) keeps the key on your laptop, but it gives the jump host access to your agent while you are logged in. Root on the jump host can then use your agent to log in to every server that trusts your key. With ProxyJump, the jump host gets nothing it can use. It only passes encrypted bytes.

When the jump host must log in by itself

Sometimes the jump host has to connect to web1 without you, for example for a nightly backup. Then make a new key on the jump host. Do not copy your own key there. In authorized_keys on web1, limit the new key:

from="198.51.100.10",restrict ssh-ed25519 AAAAC3Nz... deploy@jump cron

from= accepts the key only from the address of the jump host. restrict turns off port forwarding, agent forwarding and terminal allocation for this key. A command or an rsync job still works, but an interactive login does not. From the jump host:

$ ssh deploy@198.51.100.20 hostname
web1
$ rsync -ai deploy@198.51.100.20:site/ backup/
created directory backup
cd+++++++++ ./
>f+++++++++ index.html
>f+++++++++ style.css
$ ssh -tt deploy@198.51.100.20
PTY allocation request failed on channel 0

If this key leaks, it does not work from any other address.

My checklist

• The private key stays on the laptop, with a passphrase and ssh-agent.
• The public key goes into authorized_keys on the jump host and on the target.
• ProxyJump in ~/.ssh/config, with a Port line for the jump host when it is not 22.
• chmod 700 on ~/.ssh, and chmod 600 on the config, the private key and authorized_keys.
• On the jump host: AllowTcpForwarding local and PermitOpen.

The ssh_config and sshd_config manual pages describe every option used here.

Related reading

• rsync recipes: trailing slashes, --delete and filters
• Find which process is listening on a port in Linux
• systemd timers: a practical replacement for cron jobs
• Xtables-addons: GeoIP, TARPIT and the psd module I ported