The idea
A print studio pays its carrier for 5 Mbit/s and its uploads get half. Find where the rest goes, and win it back from the studio's side of the line.
Steps
-
Paid for five, getting two
A print studio sends proofs, the page files a printing press checks before it prints, to a press across town. The cable could carry far more, but the studio pays a carrier for 5 Mbit/s, and the carrier holds it to that.
proofDeskis the studio's computer andstudioRouterits own router. The short cable fromstudioRoutertocarrierBox, the carrier's box, is the handoff, where the studio's network meets the carrier's, and each box calls its card on that cablehandofftoo. Behind it, 20 ms away, ispressServerat the press.The press says the proofs arrive slowly. Measure the line the way anyone would: send as fast as you can for 8 seconds and see what arrives.
proofDesk:~# iperf3 -c 10.0.3.10 -t 8
-
A bucket with no queue
Carriers have a selling problem. The handoff is a cable whose cards run at 100 Mbit/s or more, and one customer pays for 5 while the next pays for 50. A different cable for every price would be absurd, so the carrier lays one kind and puts a box at its end that holds each customer to their rate. The cheapest such box is a policer: it measures what arrives and drops whatever is over. No queue, no memory to buy, and fast enough for thousands of customers at once.
A policer measures with a token bucket. Picture a bucket that fills with tokens at the paid rate: 5 Mbit/s is 625 bytes' worth every millisecond. Every packet that arrives must pay its size in tokens. If the bucket holds enough, the packet goes on and the tokens come out. If not, the packet is dropped on the spot. The bucket has a size, here 8 KB, about five full-size packets of 1,500 bytes, so a quiet minute can't be saved up for later.
The upload averages under 2.5 Mbit/s, one full packet every 5 ms or so, and in 5 ms the bucket refills over 3,000 bytes, twice what a packet costs. On average there's plenty, yet packets are lost. Watch the upload's packets leave the studio, then ask the carrier's box what it thinks of them.
studioRouter:~# tcpdump -n -q -ttt -c 12 -i handoff greater 1000 carrierBox:~# tc -s filter show dev handoff ingress
-
Shape it before they drop it
The studio can't touch the carrier's box. But it owns
studioRouter, the last box before the handoff, and it can choose how its packets leave. If they left one at a time, spaced so that each arrived after the carrier's bucket had refilled enough to pay for it, the bucket would never run dry and the carrier would drop nothing.That's a shaper. It uses the same token bucket as the policer, with one difference: when a packet comes and the bucket is short, the shaper doesn't drop it. It holds the packet in a queue until enough tokens have dripped in, then sends it. Clumps go in, an even stream comes out.
Linux's simplest shaper is
tbf, the token bucket filter. Despite its name it isn't a filter like the carrier's policer, which picks packets out and acts on them as they pass. It's a queueing rule, one that holds packets, and it takes three settings:rate 4800kbit, a little under the 5 Mbit/s paid for, so small slips in timing never put a packet ahead of the carrier's tokens. Intc,kbitis kilobits a second andkbpsis kilobytes a second, eight times as much, a classic mix-up.burst 6k, the shaper's own bucket, 6 KB. Whenever that bucket is full, tbf lets 6 KB go back to back, so it has to fit in the carrier's 8 KB.latency 40ms, about the longest a packet may wait in tbf's queue:tcturns it into the queue's size, and tbf drops what doesn't fit.
Put the shaper on, measure the line again, and watch how the packets leave now.
studioRouter:~# tc qdisc add dev handoff root tbf rate 4800kbit burst 6k latency 40ms proofDesk:~# iperf3 -c 10.0.3.10 -t 8 studioRouter:~# tc -s qdisc show dev handoff studioRouter:~# tcpdump -n -q -ttt -c 12 -i handoff greater 1000
-
The price is a queue
A shaper doesn't make the line faster. It moves the wait. Packets the carrier would have dropped wait in
studioRouter's queue instead, and anything else leaving the studio waits behind them: a video call, a web page, a ping.How long can that wait get? That's tbf's
latency.tcturns it into a queue size: 4.8 Mbit/s for 40 ms is 24,000 bytes, plus the 6 KB burst, about 20 full packets. A packet that finds the queue full is dropped.Dropped sounds bad. A dropped packet is a wasted one, the reasoning goes, so give the queue more room and nothing need be dropped. As memory got cheap, the queues in modems and routers grew on that reasoning. Put it to the test: ping the press while the upload runs, then give tbf ten times the room, 400 ms, and ping again.
proofDesk:~# ping -c 5 10.0.3.10 studioRouter:~# tc qdisc change dev handoff root tbf rate 4800kbit burst 6k latency 400ms proofDesk:~# ping -c 5 10.0.3.10 studioRouter:~# tc -s qdisc show dev handoff
-
The carrier's chair
Now sit in the carrier's chair. Its policer works: nobody gets more than they pay for. But an 8 KB bucket punishes every customer who runs TCP, which is nearly everyone, and the support calls are piling up.
Why doesn't the carrier shape instead? A shaper needs a queue, and a card queues only what it sends. On the handoff the carrier's card only receives the studio's packets, so all it can do there is measure them. Further in, the carrier's box does send them on, but on cards that carry thousands of customers at once, where shaping each one would take a queue per customer and a timer spacing out every packet. A policer needs only a count of tokens per customer, the low cost that made it the carrier's tool in the first place. So the carrier does the cheap thing: it makes the bucket bigger.
A bucket of 64 KB holds about 100 ms of the paid rate, since 5 Mbit/s for 0.1 s is 62,500 bytes, and a clump of 40 full packets fits in it. The average still can't pass 5 Mbit/s, because tokens still drip in at that rate. Only the clumps get room.
The lab has taken the studio's shaper off, so the clumps are back. This matchall filter can't be changed in place, so on
carrierBoxdelete it and add it again with the bigger bucket. Then measure the line, and ping the press while the upload runs.carrierBox:~# tc filter del dev handoff ingress carrierBox:~# tc filter add dev handoff ingress matchall action police rate 5mbit burst 64k drop proofDesk:~# iperf3 -c 10.0.3.10 -t 8 proofDesk:~# ping -c 5 10.0.3.10
Run these steps on real machines