Labs / Reliability & Traffic / Speed & queues

Lab 1 of 5 · Speed & queues

QoS queues: lag on tournament night

Lag a whole tournament with one upload, then slow the line down to fix it.

Start this lab 5 steps

The idea

A gaming café's line has room for fifty players at once. What can one upload do to them?

A full buffer is a wait
A full buffer is a wait
The mark is reset on the way
The mark is reset on the way
The queue moves home
The queue moves home
A small queue for each conversation
A small queue for each conversation
A mark counts only from a trusted machine
A mark counts only from a trusted machine

Steps

  1. Lag on tournament night

    Friday is tournament night at the café, and every player's game talks to a server in town over one 2 Mbit/s line. A game sends very little, a few dozen small updates a second, about 35 kbit/s, so on paper the line has room for fifty players. Tonight the café is also uploading last night's matches to a video site, over the same line.

    gamingPc is a player's machine and uploadBox holds the recordings. Both go through cafeRouter, the café's own router, to lineBox, the box the provider put in, and from there down the line. Play three seconds on a quiet line first, so you know what smooth looks like. Then start the upload and play again.

    gamingPc:~# gameping 3
    uploadBox:~# start-upload
    gamingPc:~# gameping 5
    lineBox:~# tc -s qdisc show dev town
  2. A mark nobody reads

    The game has been doing something all along: every update it sends carries a label asking for better treatment. Putting some traffic ahead of other traffic is called quality of service, or QoS, and the label is how a packet asks for it.

    The label lives in the IP header. The header has had a byte for this since 1981, and for years most routers ignored it. In 1998 it was rewritten: its first six bits became a DSCP number, naming a class of traffic, and the last two were later given a separate job, warning of congestion, and are 0 here. The classes are agreements, collected in RFC 4594, a guide networks follow if they choose to: 46, called EF, for voice; 32, called CS4, for real-time interactive traffic such as games; and 0 for everything else.

    The game marks its updates CS4. tcpdump prints the whole byte as tos, short for type of service, its 1981 name. 32 in binary is 100000, CS4's six bits. Add the two bits after them, both 0, and the byte is 1000 0000, which tcpdump writes in hex, base 16, as 0x80. The updates are UDP packets to the game server's port 27015. Catch two as they leave the café, then two more as they reach the game server.

    cafeRouter:~# tcpdump -n -v -c 2 -i floor udp dst port 27015
    gameServer:~# tcpdump -n -v -c 2 -i town udp dst port 27015
    lineBox:~# iptables -t mangle -S FORWARD
  3. Slower on purpose

    cafeRouter can't put the game first yet, and the reason is simple: it never has more than one packet to choose from. Its card to lineBox, handoff, runs far faster than the line, so every packet leaves the moment it arrives, and the waiting happens one box later, in a queue the café doesn't control. Putting the game first takes a queue to put it first in. The classic Linux guide in the references puts it plainly: you have to own the queue, which means being the slowest link in the chain.

    So the café makes itself the slowest link: cafeRouter sends at 1.9 Mbit/s, a little under the line's 2. Holding traffic to a rate like that is called shaping. The mark is only wiped at lineBox, one box later, so cafeRouter still sees it. The plan in /etc/queues/cafe.tc uses HTB, Linux's hierarchical token bucket, which shapes and also splits the rate into classes:

    • 1:1 is the whole café, at 1900 kbit/s.
    • 1:10 is the game: 300 kbit/s guaranteed, more when the rest of the café isn't using it, and prio 0, so while it stays within its 300 kbit/s it goes ahead of 1:20, and it gets first pick of any spare.
    • 1:20 is everything else, the default 20 on the first line, with 1600 kbit/s guaranteed.
    • The filter sends a packet to 1:10 when its mark's byte, the one tcpdump prints as tos, is 0x80. The mask 0xfc is 1111 1100, so it compares the six DSCP bits and ignores the last two.
    cafeRouter:~# cat /etc/queues/cafe.tc
    cafeRouter:~# tc -batch /etc/queues/cafe.tc
    gamingPc:~# gameping 5
    cafeRouter:~# tc -s class show dev handoff
    lineBox:~# tc -s qdisc show dev town
  4. Everyone else waits

    The game is fixed, and it was never the only small thing on the line. A café also has a customer loading a web page, the payment terminal at the bar asking the bank to approve a coffee, and every phone looking up names. None of it is marked, so all of it lands in 1:20 with the upload.

    Each class holds a queue of its own, and you didn't choose one, so each got the default: a plain line of up to 1000 packets, first in, first out. The 1000 is the card's default queue length, and it's ten times the line's whole buffer. Try a ping during the upload.

    The fix is a different kind of queue inside 1:20. SFQ, stochastic fairness queueing, tells conversations apart by their addresses and ports, gives each busy one a small queue of its own, and takes a packet's worth from each in turn. A heavy conversation then fills only its own little queue. Stochastic, here, means the sorting is a scramble of addresses and ports into 1024 buckets, so now and then two conversations land in the same bucket and share a queue.

    gamingPc:~# ping -c 4 10.0.3.10
    cafeRouter:~# tc qdisc add dev handoff parent 1:20 sfq
    gamingPc:~# ping -c 4 10.0.3.10
    cafeRouter:~# tc -s qdisc show dev handoff
  5. Everybody's a gamer

    A mark is a request, and anyone can make one. Say the upload tool on uploadBox has a turbo mode, and what turbo does is put the game's mark, CS4, on the upload's own packets. cafeRouter's filter reads the mark and nothing else, so it can't tell a game update from an upload packet wearing the same one.

    Networks deal with this by deciding where marks are believed. That place is called the trust boundary: past it a mark counts, and at it a mark from anyone the network doesn't trust is reset. In an office it's usually the switch port, which trusts the desk phone and not the laptop plugged in beside it. In the café it's cafeRouter, on the way in from the floor.

    iptables can rewrite a packet as it arrives. Its mangle table is for rules that change packets, and its PREROUTING chain sees each packet as it arrives, before the router decides where it goes or queues it to go out. Turn turbo on, play, and catch an upload packet to see its mark. Then set the boundary: every packet from the floor that isn't from gamingPc gets its mark reset to 0.

    uploadBox:~# start-upload turbo
    gamingPc:~# gameping 5
    cafeRouter:~# tcpdump -n -v -c 2 -i floor src 10.0.1.20
    cafeRouter:~# iptables -t mangle -A PREROUTING -i floor ! -s 10.0.1.10 -j DSCP --set-dscp 0
    gamingPc:~# gameping 5

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab