Labs / Troubleshooting / Where it breaks

Lab 4 of 4 · Where it breaks

What's listening? Tell a refused port from a blocked one, and find who holds it

Check open ports, Linux side and network side: closed or blocked, and who holds each.

Start this lab 5 steps

The idea

The door tablet at Lumen Gym checks members in on a web page that gymServer serves, and all week it keeps failing, a different way each day. Learn to tell the failures apart by what they say, and by how long they take to say it.

A knock on an empty port is answered at once
A knock on an empty port is answered at once
The page listens on loopback only
The page listens on loopback only
The rule drops the knock before nginx sees it
The rule drops the knock before nginx sees it
Three things a knock can get back
Three things a knock can get back
The first program to ask keeps the port
The first program to ask keeps the port

Steps

  1. A no that comes at once

    Monday, 6:00. The gym opens, and the door tablet can't load the check-in page. Over the weekend the vendor installed it on gymServer, and their setup sheet says port 8080.

    An address gets a packet to a machine, and a Port gets it to one program on that machine. A program that wants visitors asks the machine to listen on a port for it, here a TCP port, the kind web pages use. Only then can anything connect.

    When a knock arrives at a port nobody listens on, the machine answers it with a reset: nobody here. That's what connection refused means.

    So try the page from the tablet, then ask the server what it listens on. ss lists it, and -p adds the program behind each port.

    doorTablet:~# curl -sS -m 5 http://10.0.1.20:8080/
    gymServer:~# ss -tlnp
    doorTablet:~# curl -sS -m 5 http://10.0.1.20:8080/
  2. Listening, but only to itself

    Tuesday. The vendor pushed a security update to gymServer overnight, tested the page on the server, and it loaded. The tablet still can't check anyone in.

    A server has more than one address. 10.0.1.20 is its address on the gym's network. 127.0.0.1 is its loopback address, which always means this machine and never leaves it.

    In nginx's config, listen 8080; takes every address the machine has, and listen 127.0.0.1:8080; takes loopback alone. In ss, the first shows as 0.0.0.0:8080, the second as 127.0.0.1:8080.

    So it works on the server proves less than it sounds.

    doorTablet:~# curl -sS -m 5 http://10.0.1.20:8080/
    gymServer:~# curl -sS -m 5 http://127.0.0.1:8080/
    gymServer:~# ss -tlnp
    doorTablet:~# curl -sS -m 5 http://10.0.1.20:8080/
  3. Silence means something dropped it

    Wednesday. IT came in to tidy gymServer's Firewall rules, and now the tablet hangs on a blank screen for a while before it fails.

    A firewall checks every packet that arrives against its list of rules, before any program sees it. A rule can let the packet in, or drop it: throw it away and say nothing.

    That silence changes what the tablet sees. With no answer, it can't tell a dropped knock from a lost one, so its TCP sends the knock again, a few times, while curl waits out its own timer. A long wait is the sign that something got in the way and kept quiet.

    The gym's firewall has one job: stopping devices on the gym floor from reaching the admin page on 8088. iptables -L INPUT lists the rules for packets arriving at gymServer, -v adds how many packets each rule has caught, and --line-numbers numbers them.

    doorTablet:~# curl -sS -m 5 http://10.0.1.20:8080/
    gymServer:~# ss -tlnp
    gymServer:~# iptables -L INPUT -n -v --line-numbers
    gymServer:~# iptables -R INPUT 1 -p tcp --dport 8088 -j DROP
    doorTablet:~# curl -sS -m 5 http://10.0.1.20:8080/
  4. One scan, three answers

    Three days, three kinds of answer: refused at once, silence until the timer ran out, and the page itself. Those are the three you'll meet most often when you knock on a TCP port.

    nmap knocks on many ports at once and sorts what comes back into the same three:

    • open: a program accepted the knock.
    • closed: the machine answered no, nobody here.
    • filtered: no usable answer, silence or an ICMP unreachable, so something got in the way.

    Scan three ports: 80, where web pages usually live but nothing should listen at the gym, 8080 for check-in, and 8088 for the admin page. -n skips looking up names, and -p lists the ports to try.

    doorTablet:~# nmap -n -p 80,8080,8088 10.0.1.20
    doorTablet:~# nmap -n -sV -p 8080 10.0.1.20
  5. Open, but the wrong program

    Friday, 6:00, after a power cut in the night. The tablet shows a page, only not the check-in one.

    A port takes one listener, and whichever program asks for it first gets it. So when something unexpected answers on your port, the program you wanted may have lost the race. Asking for an address and port is called binding to it.

    ss -p names the program holding a port, with its pid, the number the system gave that running program. kill and a pid stop that one program.

    doorTablet:~# curl -sS -m 5 http://10.0.1.20:8080/
    gymServer:~# nginx -c /etc/nginx/checkin.conf
    gymServer:~# ss -tlnp
    gymServer:~# pkill -x python3
    gymServer:~# nginx -c /etc/nginx/checkin.conf
    doorTablet:~# curl -sS -m 5 http://10.0.1.20:8080/

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab