Labs / Reliability & Traffic / Speed & queues

Lab 2 of 5 ยท Speed & queues

Traffic shaping: paid for five, getting two

Lose half a paid-for line to the carrier's policer, then shape to win it back.

Start this lab 5 steps

The idea

A print studio pays its carrier for 5 Mbit/s and its uploads get half. Find where the rest goes, and win it back from the studio's side of the line.

Half of it arrives
Half of it arrives
The sixth packet
The sixth packet
The bucket and the queue
The bucket and the queue
The ping waits at the back
The ping waits at the back
A bigger bucket, the same rate
A bigger bucket, the same rate

Steps

  1. Paid for five, getting two

    A print studio sends proofs, the page files a printing press checks before it prints, to a press across town. The cable could carry far more, but the studio pays a carrier for 5 Mbit/s, and the carrier holds it to that.

    proofDesk is the studio's computer and studioRouter its own router. The short cable from studioRouter to carrierBox, the carrier's box, is the handoff, where the studio's network meets the carrier's, and each box calls its card on that cable handoff too. Behind it, 20 ms away, is pressServer at the press.

    The press says the proofs arrive slowly. Measure the line the way anyone would: send as fast as you can for 8 seconds and see what arrives.

    proofDesk:~# iperf3 -c 10.0.3.10 -t 8
  2. A bucket with no queue

    Carriers have a selling problem. The handoff is a cable whose cards run at 100 Mbit/s or more, and one customer pays for 5 while the next pays for 50. A different cable for every price would be absurd, so the carrier lays one kind and puts a box at its end that holds each customer to their rate. The cheapest such box is a policer: it measures what arrives and drops whatever is over. No queue, no memory to buy, and fast enough for thousands of customers at once.

    A policer measures with a token bucket. Picture a bucket that fills with tokens at the paid rate: 5 Mbit/s is 625 bytes' worth every millisecond. Every packet that arrives must pay its size in tokens. If the bucket holds enough, the packet goes on and the tokens come out. If not, the packet is dropped on the spot. The bucket has a size, here 8 KB, about five full-size packets of 1,500 bytes, so a quiet minute can't be saved up for later.

    The upload averages under 2.5 Mbit/s, one full packet every 5 ms or so, and in 5 ms the bucket refills over 3,000 bytes, twice what a packet costs. On average there's plenty, yet packets are lost. Watch the upload's packets leave the studio, then ask the carrier's box what it thinks of them.

    studioRouter:~# tcpdump -n -q -ttt -c 12 -i handoff greater 1000
    carrierBox:~# tc -s filter show dev handoff ingress
  3. Shape it before they drop it

    The studio can't touch the carrier's box. But it owns studioRouter, the last box before the handoff, and it can choose how its packets leave. If they left one at a time, spaced so that each arrived after the carrier's bucket had refilled enough to pay for it, the bucket would never run dry and the carrier would drop nothing.

    That's a shaper. It uses the same token bucket as the policer, with one difference: when a packet comes and the bucket is short, the shaper doesn't drop it. It holds the packet in a queue until enough tokens have dripped in, then sends it. Clumps go in, an even stream comes out.

    Linux's simplest shaper is tbf, the token bucket filter. Despite its name it isn't a filter like the carrier's policer, which picks packets out and acts on them as they pass. It's a queueing rule, one that holds packets, and it takes three settings:

    • rate 4800kbit, a little under the 5 Mbit/s paid for, so small slips in timing never put a packet ahead of the carrier's tokens. In tc, kbit is kilobits a second and kbps is kilobytes a second, eight times as much, a classic mix-up.
    • burst 6k, the shaper's own bucket, 6 KB. Whenever that bucket is full, tbf lets 6 KB go back to back, so it has to fit in the carrier's 8 KB.
    • latency 40ms, about the longest a packet may wait in tbf's queue: tc turns it into the queue's size, and tbf drops what doesn't fit.

    Put the shaper on, measure the line again, and watch how the packets leave now.

    studioRouter:~# tc qdisc add dev handoff root tbf rate 4800kbit burst 6k latency 40ms
    proofDesk:~# iperf3 -c 10.0.3.10 -t 8
    studioRouter:~# tc -s qdisc show dev handoff
    studioRouter:~# tcpdump -n -q -ttt -c 12 -i handoff greater 1000
  4. The price is a queue

    A shaper doesn't make the line faster. It moves the wait. Packets the carrier would have dropped wait in studioRouter's queue instead, and anything else leaving the studio waits behind them: a video call, a web page, a ping.

    How long can that wait get? That's tbf's latency. tc turns it into a queue size: 4.8 Mbit/s for 40 ms is 24,000 bytes, plus the 6 KB burst, about 20 full packets. A packet that finds the queue full is dropped.

    Dropped sounds bad. A dropped packet is a wasted one, the reasoning goes, so give the queue more room and nothing need be dropped. As memory got cheap, the queues in modems and routers grew on that reasoning. Put it to the test: ping the press while the upload runs, then give tbf ten times the room, 400 ms, and ping again.

    proofDesk:~# ping -c 5 10.0.3.10
    studioRouter:~# tc qdisc change dev handoff root tbf rate 4800kbit burst 6k latency 400ms
    proofDesk:~# ping -c 5 10.0.3.10
    studioRouter:~# tc -s qdisc show dev handoff
  5. The carrier's chair

    Now sit in the carrier's chair. Its policer works: nobody gets more than they pay for. But an 8 KB bucket punishes every customer who runs TCP, which is nearly everyone, and the support calls are piling up.

    Why doesn't the carrier shape instead? A shaper needs a queue, and a card queues only what it sends. On the handoff the carrier's card only receives the studio's packets, so all it can do there is measure them. Further in, the carrier's box does send them on, but on cards that carry thousands of customers at once, where shaping each one would take a queue per customer and a timer spacing out every packet. A policer needs only a count of tokens per customer, the low cost that made it the carrier's tool in the first place. So the carrier does the cheap thing: it makes the bucket bigger.

    A bucket of 64 KB holds about 100 ms of the paid rate, since 5 Mbit/s for 0.1 s is 62,500 bytes, and a clump of 40 full packets fits in it. The average still can't pass 5 Mbit/s, because tokens still drip in at that rate. Only the clumps get room.

    The lab has taken the studio's shaper off, so the clumps are back. This matchall filter can't be changed in place, so on carrierBox delete it and add it again with the bigger bucket. Then measure the line, and ping the press while the upload runs.

    carrierBox:~# tc filter del dev handoff ingress
    carrierBox:~# tc filter add dev handoff ingress matchall action police rate 5mbit burst 64k drop
    proofDesk:~# iperf3 -c 10.0.3.10 -t 8
    proofDesk:~# ping -c 5 10.0.3.10

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab