Labs / Routing / Provider routing

Lab 3 of 5 · Provider routing

VRF: one router, several routing tables

Split one router into several, so tenants never meet.

Start this lab 6 steps

The idea

One router serves every shop in a mall, and the shops must never see each other. You split it into several routers that share one box, then open exactly the holes the shops need.

One table joins every card
One table joins every card
A card takes its row with it
A card takes its row with it
A miss carries on, a refusal ends it
A miss carries on, a refusal ends it
One router, three tables
One router, three tables
One row, two shops with one address
One row, two shops with one address
The address changes on the cable
The address changes on the cable

Steps

  1. One router, every shop

    You run the network at a shopping mall, and one mallRouter in the basement serves every shop. The jeweler rents two networks from you: jewelerShop, 10.0.1.0/24, where jewelerTill rings up sales, and jewelerBack, 10.0.2.0/24, where jewelerLedger keeps the books. The bakery rents the same pair, bakeryShop on 10.0.3.0/24 and bakeryBack on 10.0.4.0/24.

    cardGateway, on payNet, 10.0.5.0/24, approves card payments for everybody.

    mallRouter has one network card on each of those five networks, named after the network, and in this lab "card" on its own always means that kind of card. A router adds a row to its routing table for each network on its cards, the moment a card gets an address, and then forwards packets between any two rows. That's its whole job, and it has no idea what a tenant is.

    You've set up nothing else on mallRouter. jewelerLedger is at 10.0.2.10, and the bakery has no business reading it.

    mallRouter:~# ip route
    bakeryTill:~# curl -sS -m 3 http://10.0.2.10/
  2. Split the router in two

    A router per shop would work, and providers faced that same bill in the late 1990s: one router at the edge of their network, serving many customers who must never see each other. Their answer, written up in 1999 as RFC 2547, was to split the router instead of buying more, with a separate table for each customer site.

    VRF, Virtual Routing and Forwarding, gives one router several routing tables, and each card belongs to exactly one of them. A packet is looked up in the table of the card it came in on, so two cards in different VRFs route as if they sat on two different routers, although they still share one firewall. A VLAN does the same thing one layer down, splitting a switch into several.

    Linux builds this from a fake card. ip link add jeweler type vrf table 10 makes a card named jeweler, connected to nothing, that stands for table 10, and ip link set jewelerShop master jeweler moves a real card under it. master is Linux's word for a card that owns other cards.

    You'll do this for the jeweler and then print the tables. Plain ip route prints the table you've been reading, whose name is main.

    mallRouter:~# ip link add jeweler type vrf table 10
    mallRouter:~# ip link set jeweler up
    mallRouter:~# ip link set jewelerShop master jeweler
    mallRouter:~# ip link set jewelerBack master jeweler
    mallRouter:~# ip route
    mallRouter:~# ip route show vrf jeweler
    bakeryTill:~# curl -sS -m 3 http://10.0.4.10/
    bakeryTill:~# curl -sS -m 3 http://10.0.2.10/
  3. Seal each table

    payNet hasn't moved. cardGateway belongs to no shop, so the router's payNet card stays in main, and the bakery's table has no row for 10.0.5.0/24.

    A VRF's table isn't the only one the router can read. Linux keeps all its tables in one ordered list, with the VRF's table on it and main further down, so the question is what a lookup does when it finds nothing in the VRF's table.

    You'll ping cardGateway from the bakery's till while the lab records what reaches it, then read that list, and close any gap you find.

    bakeryTill:~# ping -c 2 -W 2 10.0.5.10
    mallRouter:~# ip rule
    mallRouter:~# ip route get 10.0.5.10 vrf bakery
    mallRouter:~# ip route add unreachable default vrf bakery
    mallRouter:~# ip route add unreachable default vrf jeweler
    mallRouter:~# ip route show vrf bakery
    bakeryTill:~# ping -c 2 -W 2 10.0.5.10
  4. Share one server

    Card payments still have to work, so both shops need cardGateway, and nothing else from outside their own tables. A sealed table closes everything, so you open one hole at a time, by hand, in each direction.

    Putting a row into a table that leads out of it is called route leaking. It's on purpose here, and it has nothing to do with the accident of the same name in BGP.

    The way out is a row in the bakery's table: ip route add 10.0.5.0/24 dev payNet vrf bakery, which says this block is behind payNet, a card that isn't in this VRF. The way back is a row in main: ip route add 10.0.3.0/24 dev bakery. Its dev is the fake card, so a reply sent there is looked up in the bakery's table, which knows bakeryShop.

    You'll add both rows for the bakery, and the jeweler's two get added for you. After that, main holds a route to both shops.

    mallRouter:~# ip route add 10.0.5.0/24 dev payNet vrf bakery
    mallRouter:~# ip route add 10.0.3.0/24 dev bakery
    bakeryTill:~# curl -sS -m 3 http://10.0.5.10/
    jewelerTill:~# curl -sS -m 3 http://10.0.5.10/
    mallRouter:~# ip route
    jewelerTill:~# ping -c 2 -W 2 10.0.3.10
  5. Reuse the same numbers

    The bakery has bought a new till system, and its vendor ships it on fixed addresses: 10.0.1.0/24 for the shop and 10.0.2.0/24 for the back office. Those are the jeweler's numbers, and this happens all the time, because everyone numbers from the same private ranges. One company buys another, or two customers land on one router, and both have a 10.0.1.0/24.

    On one routing table that can't work, because only one row for a block ever gets used, and it's the second reason VRFs exist. Each VRF is its own table, so the same block can sit in both.

    The vendor has moved bakeryTill to 10.0.1.10 and bakeryLedger to 10.0.2.10, the same two addresses the jeweler uses. You'll give mallRouter the new numbers on the bakery's cards, and then fetch http://10.0.2.10/ from both shops.

    cardGateway is the catch. Its replies read main, and both shops are now 10.0.1.10. You'll try to add the bakery's way back too, and then ping cardGateway from bakeryTill with a recording running.

    mallRouter:~# ip addr add 10.0.1.2/24 dev bakeryShop
    mallRouter:~# ip addr add 10.0.2.2/24 dev bakeryBack
    mallRouter:~# ip route show vrf bakery
    mallRouter:~# ip route show vrf jeweler
    bakeryTill:~# curl -sS -m 3 http://10.0.2.10/
    jewelerTill:~# curl -sS -m 3 http://10.0.2.10/
    mallRouter:~# ip route add 10.0.1.0/24 dev bakery
    bakeryTill:~# ping -c 2 -W 2 10.0.5.10
  6. Translate at the border

    cardGateway can't tell two shops apart while both call themselves 10.0.1.10, so one of them needs a different name by the time it gets there. That's NAT: rewrite the source address on the way out, remember the change, and undo it on the reply.

    A rule on payNet wouldn't work, because the reply would get its address back in main, and main's one row for 10.0.1.0/24 still leads to the jeweler. On Linux the simplest place for NAT is a card that only the bakery's traffic crosses, so you'll build one: a veth pair, which is a virtual cable with a card at each end. ip link add bakeryLink type veth peer name mallLink makes both ends, and one end goes into the bakery's VRF while the other stays in main.

    The two tables are now joined by a link, as two real routers would be, with a small block of its own, 10.0.7.0/30, that no shop uses.

    The bakery's route to cardGateway then goes through the cable, and one rule translates everything leaving by bakeryLink.

    mallRouter:~# ip link add bakeryLink type veth peer name mallLink
    mallRouter:~# ip link set bakeryLink master bakery
    mallRouter:~# ip route replace 10.0.5.0/24 via 10.0.7.2 vrf bakery
    mallRouter:~# iptables -t nat -A POSTROUTING -o bakeryLink -j MASQUERADE
    bakeryTill:~# ping -c 2 -W 2 10.0.5.10
    bakeryTill:~# curl -sS -m 3 http://10.0.5.10/
    jewelerTill:~# curl -sS -m 3 http://10.0.5.10/

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab