SYN flood and SYN cookies

A TCP handshake is SYN, SYN-ACK, ACK. Between the first two the connection sits half-open in a fixed-size backlog. An attacker sends a flood of SYNs from forged source addresses and never completes them; the backlog fills and real clients are refused. SYN cookies are the defence built into the kernel: when the backlog overflows, the server encodes the connection's state into the SYN-ACK's sequence number and keeps no state, reconstructing it from the client's ACK — so a flood of SYNs that never ACK costs nothing. A blunt packet-rate limit also drops real traffic; the durable fix is anti-spoofing at the edge (BCP 38), which discards forged sources before they arrive.

How a SYN flood works, step by step

TCP opens with the three-way handshake: SYN, SYN-ACK, ACK. Between the first two the connection is half-open and the kernel holds it in the SYN queue, sized by net.ipv4.tcp_max_syn_backlog. A SYN flood (described as an attack and defence in RFC 4987) abuses that gap. Follow it on the SYN flood lab, where webServer at 10.0.1.10 serves a page and attacker at 10.0.1.11 floods it:

1. The flood. The attacker sends a torrent of SYNs to port 80, each with a forged source address it does not own (10.0.1.101 through 10.0.1.160). It never sends the finishing ACK.
2. The server answers. For each SYN the kernel sends a SYN-ACK to the forged address and keeps a half-open entry, retransmitting the SYN-ACK a few times (tcp_synack_retries, about 20 seconds per entry) before giving up.
3. The queue fills. The forged addresses never reply, so entries pile up. A fixed backlog would overflow and real clients would be refused: that is the denial of service.
4. SYN cookies rescue it. With net.ipv4.tcp_syncookies set, when the queue overflows the kernel stops storing state. It encodes the connection into the SYN-ACK's 32-bit sequence number (a SYN cookie: a hash of the four addresses, ports and a slow timer, plus the MSS) and allocates nothing. A forged SYN that never finishes costs no memory. Only when a real ACK comes back carrying cookie+1 does the kernel recompute it and build the socket.
5. The real fix is at the edge. The flood only works because the source is forged. Anti-spoofing (BCP 38, uRPF, or IP Source Guard on a switch) drops any packet whose source is not the address assigned to that port, so the forged SYNs never leave the attacker's link.

What it looks like on a Linux machine

Count the half-open connections on the server with ss, asking for the SYN-RECV state:

$ ss -tnH state syn-recv | head -3
0      0      10.0.1.10:80      10.0.1.137:3037
0      0      10.0.1.10:80      10.0.1.142:3042
0      0      10.0.1.10:80      10.0.1.118:3018

Before the flood, ss -tnH state syn-recv | wc -l counts 0; during it, the queue is full. Every peer is one of the forged 10.0.1.1xx addresses the attacker invented. The SYN cookie counter proves the kernel has switched over; read it with nstat:

$ nstat -az | grep -i syncookie
TcpExtSyncookiesSent            12044              0.0
TcpExtSyncookiesRecv            3                  0.0

SyncookiesSent climbs by thousands a second while SyncookiesRecv barely moves: the forged SYNs are answered with cookies but never ACKed. Through all of it a genuine client is still served, because cookies cost nothing:

$ curl -s -o /dev/null -w 'legit=%{http_code}\n' --max-time 6 http://10.0.1.10/
legit=200

A blunt server-side rate limit (iptables -A NETLAB-INPUT -p tcp --syn --dport 80 -m limit --limit 10/second -j ACCEPT then a DROP) does cut the cookie rate, but it counts forged and real SYNs in one bucket, so the real client gets legit=000. The edge filter iptables -A NETLAB-OUTPUT ! -s 10.0.1.11 -j DROP, applied at the attacker, stops the forgery at the source and leaves the real client untouched.

What goes wrong, and how you see it

SYN cookies are off. If sysctl net.ipv4.tcp_syncookies reads 0, the backlog overflows and nstat shows TcpExtListenOverflows and TcpExtListenDrops climbing while real clients see connection timeouts. Set it to 1 (the usual default).

You "fixed" it with a rate limit and locked everyone out. Real clients get legit=000 even though the server has memory to spare. A global --limit on new SYNs cannot tell forged from real; remove it and let cookies work, as the lab shows.

Cookies lose TCP options. A cookie has room only for a coarse MSS (one of a few values). Linux also packs window scaling, SACK and ECN into the TCP timestamp when the client sends one; without timestamps those options are lost for cookie-born connections, and throughput suffers while the flood lasts.

The flood is real traffic, not spoofed. A botnet completing handshakes fills accept queues instead, and cookies do not help. Here ss -tn state established is what grows; you need rate limiting or scrubbing upstream.

Questions people ask

Do SYN cookies stop every SYN flood? They defeat the spoofed, never-completed kind at no memory cost. They do not stop a flood from real addresses that finish the handshake, and they weaken TCP options. The durable fix is anti-spoofing at the network edge (BCP 38), not a server trick.

SYN flood vs a normal DoS? A SYN flood targets the handshake state (TCP connection setup) with tiny packets, so a little bandwidth exhausts a server's connection table. A volumetric DoS simply saturates the link.

Why are the source addresses fake? Forging the source (IP source spoofing) hides the attacker and guarantees no ACK comes back, keeping each entry half-open. It is exactly what edge filtering removes.

See it on real machines