Labs / Routing / Provider routing

Lab 4 of 5 · Provider routing

MPLS: a core that never reads the address

Carry customers' packets across routers that hold no route for them.

Start this lab 7 steps

The idea

A provider's routers would have to remember every customer's addresses, and that list never stops growing. You stick a small number on each packet instead, and end with two customers using the very same addresses across one set of routers.

No way on, and no way back
No way on, and no way back
Four bytes in front of the IP header
Four bytes in front of the IP header
Push, swap, pop
Push, swap, pop
A labelled packet reads one table
A labelled packet reads one table
Offers go one way, packets the other
Offers go one way, packets the other
The customer's row borrows a label
The customer's row borrows a label
The inner label names the customer
The inner label names the customer

Steps

  1. Lost in the core

    You run the network at Longline Telecom, a fiber provider whose cable follows a railway from west to east. Four routers sit along it.

    westEdge and eastEdge are at the two ends, where customers plug in, and those two are the edge. coreOne and coreTwo are in between, no customer ever touches them, and those two are the core.

    Three cables join them: linkOne, linkTwo and linkThree, west to east. Each router has a network card on every cable it touches, named after the cable, and in this lab "card" on its own means that kind of card.

    Two customers pay you. A bank has bankHq in the west on 10.0.1.0/24, and bankBranch in the east on 10.0.6.0/24, at 10.0.6.10. A radio station has radioStudio in the west on 10.0.2.0/24, and radioMast in the east on 10.0.7.0/24.

    The four cables out to those customers are bankWest, radioWest, bankEast and radioEast.

    All four routers run OSPF, so they already know the three cables and each other. Customers' networks are kept out of it, so no router has heard of them unless a customer plugs into it directly.

    You'll ping bankBranch from bankHq twice. The first try stops at westEdge, which has no row for 10.0.6.0/24 in its routing table and says so with Destination Net Unreachable. Then you add that row by hand, sending the packet on to coreOne at 10.0.12.3.

    bankHq:~# ping -c 2 -W 2 10.0.6.10
    westEdge:~# ip route add 10.0.6.0/24 via 10.0.12.3
    bankHq:~# ping -c 2 -W 2 10.0.6.10
    coreOne:~# ip route
  2. Stick a number on

    MPLS is the other way out. The first router decides which path a packet belongs to, and writes that on the packet as a small number, the label. Every router after it reads that number and nothing else.

    A label is a 4-byte header placed in front of the IP header. A router that does this keeps a second table next to its routing table, and that second table's rows are short: this label came in, send it out with that label, to this neighbour. Looking a row up is an exact match on one number, where a routing table needs the most specific block that contains the address, the one with the longest mask.

    That difference was a big selling point in the mid-1990s, when many routers did the longest match in software and it was slow.

    Toshiba, Ipsilon, Cisco and IBM all worked on label switching in those years, and an IETF working group formed in 1997 turned the ideas into Multiprotocol Label Switching, published as RFC 3031 in 2001. "Multiprotocol" because the idea works for any protocol under the label.

    Then hardware caught up, and the speed argument faded. MPLS stayed for a better reason, which you're about to see.

    You build the path from the far end, so that each router is ready before the one upstream starts sending to it. -f mpls tells ip route to work on the label table instead of the routing table.

    • On coreTwo, ip -f mpls route add 200 via inet 10.0.34.3 means: a packet with label 200 goes to eastEdge.
    • On coreOne, ip -f mpls route add 100 as 200 via inet 10.0.23.3 means: a packet with label 100 gets label 200 instead, and goes to coreTwo.
    • On westEdge, ip route replace 10.0.6.0/24 encap mpls 100 via 10.0.12.3 swaps your plain row for one that puts label 100 on first.

    A label works in one direction only, so the way back needs its own path. The lab builds that for you, with labels 300 and 400.

    coreTwo:~# ip -f mpls route add 200 via inet 10.0.34.3
    coreOne:~# ip -f mpls route add 100 as 200 via inet 10.0.23.3
    westEdge:~# ip route replace 10.0.6.0/24 encap mpls 100 via 10.0.12.3
    bankHq:~# ping -c 2 -W 2 10.0.6.10
    coreOne:~# ip route get 10.0.6.10
    coreOne:~# ip -f mpls route
  3. Watch the number change

    A row says what should happen, so now watch it happen on the cables. You send one ping, and tcpdump, which can read MPLS, records it on linkOne, linkTwo and linkThree at the same time.

    Each cable carries the request going east and the reply coming back. The reply is on its own path with its own labels, so read the echo request lines first.

    westEdge puts label 100 on, so that's what linkOne shows, and coreOne's row turns it into 200 on linkTwo. The row you gave coreTwo looks different from coreOne's: it has no as, and you gave eastEdge no label row at all.

    bankHq:~# ping -c 1 -W 2 10.0.6.10
  4. Lose one row

    So a core router can forward with no route. What about a core router that has a route and has lost its label row?

    That's a real fault. Labels and routes come from different places, so one can be missing while the other is fine.

    You'll stage it on coreOne: add a plain row for 10.0.6.0/24 towards coreTwo, then delete the row for label 100. You'll undo all of this afterwards.

    westEdge doesn't know about any of this and keeps sending label 100. You record on both of coreOne's cables: linkOne, where the packet arrives, and linkTwo, where it leaves if it goes on.

    coreOne:~# ip route add 10.0.6.0/24 via 10.0.23.3
    coreOne:~# ip -f mpls route del 100
    bankHq:~# ping -c 1 -W 2 10.0.6.10
    coreOne:~# ip -f mpls route add 100 as 200 via inet 10.0.23.3
    bankHq:~# ping -c 2 -W 2 10.0.6.10
  5. Let routers pick numbers

    You typed three rows, the lab typed three more, and that was one pair of networks. The radio station needs the same, every new customer needs the same, and any cable fault means fixing rows on routers one by one. Nobody runs a network like that.

    LDP, the Label Distribution Protocol, does this job. It runs beside OSPF and doesn't replace it: OSPF still finds the routes, and LDP goes through this router's own networks and the rows OSPF found, picks a free label for each, and tells every neighbour "for this block, send me this label". A router then installs the label its next hop offered.

    Which rows get a label is a vendor's choice, and FRR, the routing software here, leaves rows from BGP alone.

    Two LDP routers find each other with a hello, sent to every router on the cable on UDP port 646, then open one TCP connection on port 646 to trade labels. They open it between their lo addresses, the 10.0.90.x you saw in the routing table. lo is a loopback, a card with no cable, so it stays up whichever real card fails, and OSPF has been announcing each router's address on it as a /32.

    The lab clears your hand-made rows on all four routers, including the customer rows on the edges, and turns LDP on for three of them. You type the fourth, on coreOne.

    coreOne:~# vtysh -c "configure terminal" -c "mpls ldp" -c "router-id 10.0.90.2" -c "address-family ipv4" -c "discovery transport-address 10.0.90.2" -c "interface linkOne" -c "exit" -c "interface linkTwo"
    coreOne:~# vtysh -c "show mpls ldp neighbor"
    coreOne:~# vtysh -c "show mpls ldp binding"
    westEdge:~# ip route show 10.0.90.4
    bankHq:~# ping -c 2 -W 2 10.0.6.10
  6. Customer routes at the edges

    Something has to carry customer networks, and it should touch as few routers as possible. Only the two edges ever need them: westEdge must know that 10.0.6.0/24 lives behind eastEdge, and the core only has to get a packet from one edge to the other.

    That's a job for BGP. The two edges open one BGP session straight across the core, between their loopback addresses, both in the same autonomous system, 65000.

    The core routers carry that session's packets and don't run BGP. Each edge announces its customer networks, naming itself as the next hop, 10.0.90.1 or 10.0.90.4.

    You open westEdge's end. update-source lo sends from the loopback address, because that's the address eastEdge expects, and the lab opens eastEdge's end for you.

    westEdge holds no LDP label for 10.0.6.0/24, since that offer stopped at coreTwo, and all it gets from BGP is "that block's next hop is 10.0.90.4".

    westEdge:~# vtysh -c "configure terminal" -c "router bgp 65000" -c "neighbor 10.0.90.4 remote-as 65000" -c "neighbor 10.0.90.4 update-source lo" -c "address-family ipv4 unicast" -c "network 10.0.1.0/24" -c "network 10.0.2.0/24"
    westEdge:~# ip route show 10.0.6.0/24
    coreOne:~# ip route
    bankHq:~# ping -c 2 -W 2 10.0.6.10
    radioStudio:~# ping -c 2 -W 2 10.0.6.10
  7. Two labels, two customers

    That last ping is a problem. One routing table on each edge means every customer can reach every other, and the bank has noticed.

    The radio station has a second request: it has bought a transmitter company whose machines use 10.0.6.0/24, the same block as the bank's, and it isn't renumbering them. So the lab moves radioMast to 10.0.6.10, the very address bankBranch has.

    This is what providers sell as an MPLS VPN. Each edge gets one VRF per customer, a separate routing table, the same split as in the vrf-routers lab. A customer's card joins that customer's table, so bank and radio can hold the same block without a clash.

    BGP then has to carry two routes for 10.0.6.0/24 without mixing them, so it gets three additions:

    • A route distinguisher is a tag glued to the front of the block, so 65000:1:10.0.6.0/24 and 65000:2:10.0.6.0/24 are two different things to BGP.
    • A route target says which VRFs may import a route. The bank's rows are stamped 65000:1, and only the bank table takes them in.
    • A VPN label is one more label, picked by the far edge, that says what to do with the packet. FRR gives each VRF one label here, so it means "this packet belongs in that VRF".

    The lab uses one number for both tags, 65000:1 for the bank and 65000:2 for the radio station, which is common, though the two do different jobs.

    So the lab sets all of that up, and you turn it on with one command on westEdge: address-family ipv4 vpn says this session also carries VPN routes.

    A packet now leaves westEdge with two labels stacked. The inner one is the VPN label, and the outer one is LDP's, which gets the packet to the far edge exactly as before. Core routers read only the outer label, the top one.

    westEdge:~# vtysh -c "configure terminal" -c "router bgp 65000" -c "address-family ipv4 vpn" -c "neighbor 10.0.90.4 activate"
    westEdge:~# vtysh -c "show bgp ipv4 vpn"
    westEdge:~# ip route show vrf bank
    westEdge:~# ip route show vrf radio
    bankHq:~# curl -sS -m 3 http://10.0.6.10/
    radioStudio:~# curl -sS -m 3 http://10.0.6.10/
    bankHq:~# ping -c 1 -W 2 10.0.6.10
    eastEdge:~# ip -f mpls route

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab