Labs / Routing / Provider routing

Lab 5 of 5 · Provider routing

SD-WAN: every application on the right line

Steer each application over the better of two lines, from one controller.

Start this lab 6 steps

The idea

A branch with two lines to head office usually keeps one idle as a spare. SD-WAN, the software-defined WAN, puts both to work: each application rides the line that suits it, and the routers move it by themselves when that line starts losing packets.

One queue, and the ping is at the back
One queue, and the ping is at the back
A public header in front
A public header in front
The rule picks the table
The rule picks the table
One policy, rules on every edge
One policy, rules on every edge
The line is up, and the calls leave it
The line is up, and the calls leave it
Straight out, with its own NAT
Straight out, with its own NAT

Steps

  1. One line for everything

    You run the network at Kestrel Ferries. Its quay ticket office works with head office in the city: quayDesk sells tickets and takes phone calls, and quayCameras sends the day's camera footage to hqServer every evening.

    quayEdge is the quay office's router, and cityEdge is head office's. Two lines join them.

    • mplsLine is a private line rented from a provider that runs MPLS. It's steady, about 20 ms there and back, but carries only 2 Mbit/s and costs a lot.
    • broadband is an ordinary internet line from quayEdge to internetRouter, the internet provider's router. It carries ten times as much for a fraction of the price, but its times are longer and uneven, with no promises. Head office has its own internet line, cityLine, to the same router.

    Today every packet takes mplsLine, and broadband sits idle. You'll read quayEdge's routing table, ping hqServer at 10.0.4.10 from quayDesk, then start the camera backup and ping again during it.

    quayEdge:~# ip route
    quayDesk:~# ping -c 3 10.0.4.10
    quayCameras:~# iperf3 -c 10.0.4.10 -p 5201 -t 12
    quayDesk:~# ping -c 6 10.0.4.10
  2. A tunnel over broadband

    Before anything can use broadband, it has to reach head office. In this lab the addresses on broadband, cityLine and cloud play the internet's public addresses.

    The internet carries public addresses only, so internetRouter has no row for 10.0.4.0/24 or 10.0.1.0/24. You try anyway first: you give quayEdge a row for hqServer alone, 10.0.4.10/32, pointing at internetRouter, 10.0.3.3, and ping. The most specific row wins, so only hqServer moves.

    Then you give each packet a second, outer header, from quayEdge's broadband address, 10.0.3.2, to cityEdge's cityLine address, 10.0.5.3. Both are public addresses, so the internet can deliver it, and cityEdge takes the outer header off. That's a GRE tunnel, and to the router it looks like one more card, which you'll call overBroadband.

    The lab builds the far end on cityEdge, and gives the ends addresses to reach each other by, 172.16.0.1 and 172.16.0.2. Then you send hqServer's row into the tunnel, and record broadband while you ping.

    quayEdge:~# ip route replace 10.0.4.10/32 via 10.0.3.3
    quayDesk:~# ping -c 2 -W 2 10.0.4.10
    quayEdge:~# ip link add overBroadband type gre local 10.0.3.2 remote 10.0.5.3 ttl 64
    quayEdge:~# ip route replace 10.0.4.10/32 dev overBroadband
    quayDesk:~# ping -c 2 -W 2 10.0.4.10
  3. Send each application its way

    Now everything for hqServer takes broadband, and that's no better. The calls want mplsLine, short and steady, and the backup wants broadband's room. Both go to 10.0.4.10, and a routing table picks a row by destination alone.

    A Linux router can hold many routing tables: the one ip route printed is called main, and others have numbers. Above them sits ip rule, a list checked in order before any table is read. A rule can match the protocol and the port too, and send the packet to another table.

    The lab took your hqServer row away and wrote table 20, with one row: 10.0.4.0/24 into the tunnel. It did the same on cityEdge, plus a rule for the backup's replies, from port 5201: each router picks the way out for its own packets, so without it they'd come home on mplsLine.

    The backup uses TCP port 5201, and the calls use UDP port 5004. You add the backup's rule, ask quayEdge which way each would go, then rerun the backup and ping during it.

    quayEdge:~# ip rule add ipproto tcp dport 5201 lookup 20
    quayEdge:~# ip rule
    quayEdge:~# ip route get 10.0.4.10 ipproto tcp dport 5201
    quayEdge:~# ip route get 10.0.4.10 ipproto udp dport 5004
    quayCameras:~# iperf3 -c 10.0.4.10 -p 5201 -t 12
    quayDesk:~# ping -c 6 10.0.4.10
  4. Hand the rules to a controller

    Two rules on two routers suit one office. Kestrel Ferries has an office at every port it sails to, and each change would mean getting every rule right on every router.

    SD-WAN keeps the decisions in one place. A controller, here controller at 10.0.4.20, holds one policy: each application, and the path it should take. A program on every edge, the router at each end, fetches the policy every second and writes the rules itself.

    Here that program is sdwan, written for this lab and running on both edges since the start, and controller runs it too, to keep the policy. Commercial products hold an encrypted session the controller pushes changes down, and this one polls plain HTTP to stay short.

    Each edge has two paths to the other: mplsLine, and overBroadband, the tunnel. Once the policy names an application, each edge sends probes through both, five pings a second to the far edge, and sdwan paths sums up the last 25.

    The lab took away the last step's rules. On controller you add the backup, then the calls, which leave mplsLine if it loses more than 2% of the probes.

    controller:~# sdwan add backup tcp 5201 --prefer overBroadband
    controller:~# sdwan add calls udp 5004 --prefer mplsLine --max-loss 2
    cityEdge:~# ip rule
    quayEdge:~# sdwan paths
  5. A line that loses packets

    Lines rarely die outright. More often one stays up and starts losing packets, a brownout: a fault at the provider, a damp cable joint, a busy link you can't see.

    OSPF and the other routing protocols notice a line that dies. With their usual timers they go on using one that loses a packet in five, since enough of their own messages get through, and tuned to react faster they'd drop it and bring it back every minute or so. A call breaks up long before that: a few lost packets in a hundred are already audible.

    The lab now makes mplsLine drop 20% of the packets each edge sends on it, and you check the card is still up. Then you read what the programs did, make a test call with iperf3, and ping hqServer once more.

    quayEdge:~# ip link show mplsLine
    quayEdge:~# sdwan log
    quayEdge:~# ip rule
    quayDesk:~# iperf3 -c 10.0.4.10 -u -p 5004 -b 64k -l 160 -t 4
    quayDesk:~# ping -c 20 -i 0.2 10.0.4.10
    quayEdge:~# sdwan log
  6. Break out to the cloud

    Before each sailing, quayDesk checks tideService at 10.0.6.10, a tide-table service on the internet. Today that request rides mplsLine to head office, and cityEdge sends it out on cityLine behind NAT: it swaps the private sender address for its own public one, 10.0.5.3. The lab set that NAT up, as head office has always had it.

    That trip through head office is called backhaul. It spends the expensive line on traffic that never needed head office, though on the way it passes head office's firewall, which this lab leaves out. Sending it from the quay straight to the internet is local breakout.

    You trace the way the request goes today with traceroute, add a row on quayEdge sending 10.0.6.0/24 to internetRouter, and fetch the page.

    quayDesk:~# traceroute -n 10.0.6.10
    quayEdge:~# ip route add 10.0.6.0/24 via 10.0.3.3
    quayDesk:~# curl -sS -m 3 http://10.0.6.10/
    quayEdge:~# iptables -t nat -A POSTROUTING -o broadband -j MASQUERADE
    quayDesk:~# curl -sS -m 3 http://10.0.6.10/
    quayDesk:~# traceroute -n 10.0.6.10

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab