Labs / Protocols / The web & remote shells

Lab 4 of 4 · The web & remote shells

SSH: log in to a far machine and prove who's who

Log in with keys, read a host key warning, and jump through an SSH bastion.

Start this lab 7 steps

The idea

Ines looks after Radio Tide's two servers from her kitchen table. One morning her login stops with a warning in capital letters that the server may be an impostor, and she has to work out what to believe before she types her password.

Three stages, in a fixed order
Three stages, in a fixed order
The same login, read on homeRouter
The same login, read on homeRouter
One warning, two causes
One warning, two causes
The private key never travels
The private key never travels
The first value wins
The first value wins
One login inside another
One login inside another
Two machines in one option
Two machines in one option

Steps

  1. SSH: a private shell on a far machine

    Ines looks after two servers at Radio Tide, a small local station. gate faces the internet and studio sits behind it. She works from home, so she needs to type commands on machines she can't touch.

    A shell is the program that reads your commands and runs them, the one you get in any terminal. SSH, short for Secure Shell, gives you a shell on another machine over an encrypted TCP connection. It comes in two halves: ssh is the client you run, and sshd is the server program, waiting on port 22 of the far machine.

    Tatu Ylönen wrote the first version in 1995 at Helsinki University of Technology, after someone captured passwords on the university's network. The tools people used back then sent everything as plain text, and SSH was built to replace them.

    A connection goes through three stages, always in this order:

    1. TCP connects.
    2. The two sides agree on encryption keys nobody else can work out, and in that same exchange the server proves who it is.
    3. With encryption on, you prove who you are, and only then do you get a shell.

    The server proves who it is first, and you say nothing about yourself until the encryption is on. The second stage has a gap on a first visit: the laptop has never met this server, so it has nothing to check the server's proof against.

    Log in as ines to tide.lab, the name the world knows gate by.

    laptop:~# ssh ines@tide.lab
  2. Encryption: what a stranger on the path can read

    Before SSH, the usual way to reach a far shell was telnet. It does the same job with one difference: everything crosses the network as plain text. gate still answers telnet on port 23, so compare the two side by side.

    Every packet between the laptop and gate passes through homeRouter. On a real network that spot belongs to whoever runs the router: your internet provider, a cafe's owner, or someone who has broken in.

    Nobody has to break any code to read a passing packet. They only need to be on the path and run a capture, the same one you'll run here.

    Start tcpdump on homeRouter, log in over telnet, and stop the capture. Then log in over SSH, with a command that starts a capture of its own on port 22, shown under its output.

    The SSH command uses sshpass, a helper that types the password for you, so one command does the whole login. It's a lab convenience, and it leaves the password in your command history, so keep it off real machines.

    homeRouter:~# tcpdump -l -nn -i home -A 'dst port 23 and tcp[tcpflags] & tcp-push > 0'
    laptop:~# telnet tide.lab
    laptop:~# sshpass -p tide-4 ssh ines@tide.lab hostname
  3. Host keys: the server proves who it is

    Encryption alone doesn't help if you're talking to the wrong machine. Someone on the path could answer in gate's place, encrypt everything with you, and read your password on arrival. So the server has to prove who it is before you say anything.

    Each server has a host key: a pair of matching keys, one public and one private, made when SSH is installed and kept in /etc/ssh. The public half goes out to anyone who connects. The private half never leaves the machine, and the server uses it to sign each connection.

    A signature is a value that only the holder of the private half can make, and that anyone with the public half can check. Your laptop checks the signature with that public half, then looks for the key in known_hosts, the file ssh wrote during your first login. Three things can happen:

    • The key is on the list: it's the same machine as last time, so ssh goes on.
    • No entry for that name: it's a first visit, so ssh asks you.
    • A different key: that's the warning Ines got this morning.

    Here's the catch: a rebuilt server has a new key, and an attacker in the middle has a key of its own, and to your laptop the two look identical. The warning says the key changed and can't say why, so the answer has to come from somewhere outside that connection.

    laptop:~# ssh ines@tide.lab hostname
    gate:~# ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
    laptop:~# ssh-keygen -R tide.lab
  4. Key login: prove who you are without sending a secret

    A password has one big weakness: to check it, the server has to receive it. It arrives encrypted, and then sshd holds it in plain form. A machine that has been taken over, or a wrong machine you said yes to, gets the real thing.

    A key login never sends a secret. You make a pair of keys on your laptop, and they only work together: the private key stays on the laptop, and the public key goes on the server, as one line in ~/.ssh/authorized_keys of the account you log in to. The public half is safe to hand out, since it can check a signature and can't make one.

    At login the laptop names the public key, and the server says whether it knows it. The laptop then signs a block of data that includes a value unique to this connection, and the server checks the signature with the public key. Nothing reusable crosses the network, so a recorded login is worth nothing to anyone.

    ssh-keygen makes the pair, and -t ed25519 picks Ed25519, a kind with short keys.

    ssh-copy-id logs in once with your password and adds your public key to the right file with the right permissions. It will ask about the machine's key first, since you made the laptop forget it, so check that fingerprint against the one you read on gate.

    laptop:~# ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519
    laptop:~# ssh-copy-id ines@tide.lab
    laptop:~# ssh -v ines@tide.lab hostname
  5. Passwords off: stop answering the guessers

    A machine with port 22 open to the internet starts getting login attempts soon after it goes online. Programs walk through every address they can find, trying common names and common passwords, and they never get tired. bot plays that part here.

    Now that Ines has a key, a password on gate gives her nothing and gives the guessers something to try. The fix is to turn password logins off, and the order matters.

    Prove the key works first, as you did, and keep one logged-in session open while you change the settings. A reload leaves open sessions alone, so if the change goes wrong you still have a way in to undo it.

    sshd reads its settings from /etc/ssh/sshd_config, and a line near the top of that file pulls in every .conf file in /etc/ssh/sshd_config.d/, in name order. For most settings, sshd keeps the first value it finds and ignores the rest. That surprises people: a line added to the end of the main file can do nothing at all, because a file in that folder got there first.

    Some server images from cloud providers ship such a file with passwords on, and this lab has one called 50-lab.conf. So name your file 10-keys-only.conf and it wins.

    Two settings matter, since there are two ways to ask for a password. PasswordAuthentication is the direct one, and KbdInteractiveAuthentication is a general question-and-answer login that usually ends up asking the same thing.

    gate:~# grep 'Failed password' /var/log/auth.log
    bot:~# ssh ines@tide.lab hostname
    laptop:~# ssh ines@tide.lab hostname
  6. Jump host: reach a machine with no way in

    studio runs the on-air system, so it has no address the internet can reach. gate has one foot on each side: one address on the internet network and one on station. A machine used this way, as the single guarded door to the private machines behind it, is called a jump host or SSH bastion.

    The obvious way through is two logins: SSH to gate, then SSH again from there. That works, and it has a cost.

    The second login starts on gate, so your private key has to be there too, or gate has to borrow it. A machine facing the internet is the worst place to keep one, and whoever controls it can read that second session.

    ssh -J does it differently. The laptop logs in to gate and asks it for one thing: open a TCP connection to port 22 on studio, and pass the bytes along. The laptop then runs a whole second SSH connection through that pipe, straight to studio.

    Your private key signs both logins from the laptop, and the inner connection is encrypted from end to end, so gate carries bytes it can't read.

    The target's name is looked up on gate, not on the laptop, and that's handy, since private names often mean nothing outside. The laptop's key has to be on studio as well, so you'll see that copied for you.

    Typing -J every time gets old, so you'll finish by saving the jump in ~/.ssh/config, the file where ssh keeps settings for each machine.

    laptop:~# ssh -o ConnectTimeout=3 ines@studio.lab hostname
    laptop:~# ssh -J ines@tide.lab ines@studio.lab hostname
    studio:~# grep Accepted /var/log/auth.log
  7. Port forwarding: carry one port through SSH

    studio runs a small status page for the people in the building. It shows the on-air state and the transmitter, and nothing protects it, so it listens only on 127.0.0.1, port 9000.

    That address means "this same machine", so the page can't be reached from any network, and that keeps it safe. It also keeps Ines out, from her kitchen.

    SSH can carry other connections inside its own, and this is called port forwarding, or a tunnel. With -L, the laptop's ssh starts listening on a port of the laptop. Each connection to that port goes through the SSH connection, and at the far end sshd makes the matching connection for you.

    The option has three parts: -L 9000:127.0.0.1:9000. The first 9000 is the port to open on the laptop, and the 127.0.0.1:9000 after it is the address to connect to from the far end, so it means studio. That one catches everybody out, since the same 127.0.0.1 on the laptop would mean the laptop, and a name in that spot would be looked up on studio too.

    First check the page really is closed, from gate, the closest machine there is. Then open the tunnel, with -N to say you want no shell, only the forwarding.

    The ssh ines@studio.lab part uses the jump you saved, so this tunnel runs inside the two logins you already have. With the tunnel up, the page is at a local address on the laptop.

    gate:~# curl -sS -m 3 http://studio.lab:9000/
    laptop:~# ssh -N -L 9000:127.0.0.1:9000 ines@studio.lab
    laptop:~# curl -s http://127.0.0.1:9000/

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab