The idea
Two machines in different halls want to sit on one network, and only routers join the halls. VXLAN carries each frame inside an ordinary packet, which builds an overlay, a network that exists only at its two ends.
Steps
-
Two halves of one network
Kestrel Hosting rents out machines in two halls, north and south. Each hall has an edge Router,
northEdgeandsouthEdge, and a third router,backbone, joins them. Everything between those routers is ordinary IP routing, and Kestrel likes it that way: a routed network finds a way round a cut cable, and trouble in one hall stays in that hall.Marlow Books rents a machine in each hall,
marlowOnein the north andmarlowTwoin the south, and wants both on one network,192.168.50.0/24. Customers ask for this all the time. Some software, like a pair of databases copying to each other, expects its machines to be neighbours. A machine that moves between halls has to keep its IP address, or everything pointing at it breaks. And the addresses are Marlow's own choice, so Kestrel's routers have never heard of them.Marlow is asking for one broadcast domain stretched across two halls. When
marlowOnewantsmarlowTwo, it sees an address on its own network, so it doesn't look for a gateway. It needs the other machine's hardware address, and it asks with an ARP broadcast, a question to everyone on its cable.The old way round was to run a VLAN through switches all the way from one hall to the other. It works, and it gives up everything the routers were giving Kestrel: one loop in a stretched network floods both halls at once, and there are only 4,094 VLAN numbers to share between all of Kestrel's customers.
So start with what's there now. Each machine's card is set by hand, to
192.168.50.11and192.168.50.12, and its cable runs to its hall's edge router. Ping across and see.marlowOne:~# ping -c 2 -W 1 192.168.50.12 marlowOne:~# ip neigh show 192.168.50.12 backbone:~# ip route
-
Wrap the frame in a packet
The routers will carry any packet addressed to something they know, and that's the way through. A frame is what travels on a cable: a packet with an Ethernet header on the front, which holds the hardware addresses. Suppose
northEdgetakes Marlow's whole frame, ARP question and all, and puts it inside an ordinary packet addressed tosouthEdge. The routers carry that packet across without looking inside.southEdgetakes the frame back out and puts it on Marlow's south cable, as if the two cables were one.A path like that is a tunnel, and the network the customer sees through it is the overlay. VXLAN, Virtual Extensible LAN, is the standard way to build one for Ethernet, published in 2014 for exactly Kestrel's problem. Each end of a tunnel is a VTEP, a VXLAN tunnel endpoint, and here that's
northEdgeandsouthEdge. The wrapping is an ordinary UDP packet to port 4789 with a small VXLAN header inside it, and that header's main job is one number, the VNI, the VXLAN network identifier, which says whose network this frame belongs to. Marlow gets VNI 100.On Linux, the tunnel end is a card of type
vxlan. Whatever goes out of that card is wrapped and sent to the far end, and whatever arrives on port 4789 comes out of it unwrapped. That card still has to be joined tomarlowNorth, the card Marlow's cable plugs into, and joining two cards without routing is a Switch's job. On Linux, a switch made of software is called a bridge. One machine can do both jobs:northEdgestays a router on its backbone side, and on Marlow's side it becomes a switch.So
northEdgeneeds three things: a bridge,br100, a tunnel card in that bridge,vxlan100, andmarlowNorthin the bridge as well. The tunnel card's command is the long one:ip link add vxlan100 master br100 up type vxlan id 100 remote 10.0.6.2 dstport 4789 dev northLinkmaster br100plugs the new card into the bridge, andupturns it on.type vxlan id 100makes it a tunnel card for VNI 100.remote 10.0.6.2issouthEdge, the far end of the tunnel.dstport 4789is the standard UDP port.dev northLinkis the card the wrapped packets leave by.
southEdgeneeds the mirror image, with10.0.5.2as its far end andsouthLinkas its way out. The lab does its bridge and its cable, and you do the tunnel card.Then look at one number. Every Router takes one off a packet's TTL as it passes it on, so the TTL on a reply normally shows how many routers it crossed.
northEdge:~# ip link add br100 up type bridge northEdge:~# ip link add vxlan100 master br100 up type vxlan id 100 remote 10.0.6.2 dstport 4789 dev northLink northEdge:~# ip link set marlowNorth master br100 southEdge:~# ip link add vxlan100 master br100 up type vxlan id 100 remote 10.0.5.2 dstport 4789 dev southLink marlowOne:~# ping -c 2 -W 1 192.168.50.12 marlowOne:~# ip neigh show 192.168.50.12
-
Read what the backbone carries
Marlow's ping now crosses
backbone, a router with no route to192.168.50.0/24, and that's worth a look. What does it actually see? Marlow's manager has a sharper question: other customers' traffic crosses that backbone too, so could someone there read the shop's?A capture on
backbone's cable to the north hall answers both.marlowOne:~# ping -c 2 -W 1 192.168.50.12 backbone:~# ip route get 192.168.50.12
-
Fifty bytes go missing
Marlow moves the shop onto the tunnel. Its home page is small, and
marlowTwoalso serves the stock list, about 13 KB, which the staff open all day.To think about what happens to that list, you need the MTU, the biggest packet a card will send. Nearly every Ethernet card in the world says 1500 bytes, and every card at Kestrel started out that way. TCP cuts anything long into pieces, and the two ends settle the size of a piece when they connect, from what their own cards say. Both say 1500 here, so the list goes out as about ten packets of 1500 bytes.
Your capture showed every packet growing on its way through, from
length 98tolength 148.A packet too big for the next cable normally has two ways out. A router can cut it into fragments, or, if the packet is marked "don't fragment", as TCP's are, the router sends back an ICMP message saying how big it can be. The sender then shrinks its pieces, and that's path MTU discovery. Both of those are a router's work.
Fetch both pages, then measure.
ping -M domarks a ping "don't fragment", and-ssets how many bytes of data it carries, so a ping of-s 1472plus 28 bytes of its own headers is exactly 1500 bytes.marlowOne:~# curl -sS -m 3 192.168.50.12 marlowOne:~# curl -sS -m 3 -o stock.txt 192.168.50.12/stock marlowOne:~# ping -c 1 -W 1 -M do -s 1472 192.168.50.12 northEdge:~# ip link show dev vxlan100 marlowOne:~# ip link show dev eth0 marlowOne:~# ping -c 1 -W 1 -M do -s 1422 192.168.50.12 marlowOne:~# ip link set dev eth0 mtu 1450 marlowTwo:~# ip link set dev eth0 mtu 1450 marlowOne:~# curl -sS -m 3 -o stock.txt 192.168.50.12/stock marlowOne:~# wc -l stock.txt marlowOne:~# ping -c 1 -W 1 -M do -s 1472 192.168.50.12
-
A second customer with the same addresses
Fennel Foods signs up, with a machine in each hall,
fennelOneandfennelTwo. Fennel's staff chose their own addresses, and they chose the same ones as Marlow:192.168.50.11and192.168.50.12.On a routed network that would be a disaster, because a routing table can't send one address to two places. So Fennel gets a bridge of its own and a tunnel card of its own, with VNI 200. Everything else stays the same: both customers' tunnels run between the same two edges on the same UDP port, and on the backbone the only thing telling them apart is that one number in the VXLAN header.
The number is 24 bits long, which gives about 16 million networks, against a VLAN number's 12 bits. That's the Extensible in the name, and a big hosting company needs it, which was half the reason VXLAN was written.
Build the north side yourself this time, three commands in the same shape as before, with
200for100and Fennel's card,fennelNorth. The lab does the south side.northEdge:~# ip link add br200 up type bridge northEdge:~# ip link add vxlan200 master br200 up type vxlan id 200 remote 10.0.6.2 dstport 4789 dev northLink northEdge:~# ip link set fennelNorth master br200 fennelOne:~# curl -sS -m 3 192.168.50.12 marlowOne:~# curl -sS -m 3 192.168.50.12 fennelOne:~# ping -c 2 -W 1 192.168.50.12
-
Open a third hall
Marlow rents a third machine,
marlowThreeat192.168.50.13, in a new hall in the east, behind a new edge router,eastEdge. That raises a question the two-hall tunnel let you skip. A frame goes intovxlan100, so how does the card know which hall to send it to?It keeps a table, the way a Switch does, called the forwarding database, or fdb, and
bridge fdb showprints it. Here are three of its lines onnorthEdge, two aboutmarlowTwo's hardware address and one that's all zeros:8e:10:ba:0d:ed:89 master br100 8e:10:ba:0d:ed:89 dst 10.0.6.2 self 00:00:00:00:00:00 dst 10.0.6.2 via eth1 self permanentA line with
master br100is the bridge's, and it means that address is on one of the bridge's ports. A line withselfis the tunnel card's own, and it says which far end that address is behind. Nobody typed those in. When a frame comes out of the tunnel, the card notes its source address and which edge sent it, the same way a switch learns a port, and that's called flood and learn.The line of zeros is the flood list.
permanentmeans it was set by hand, by yourremote, so it never ages out, and the card afterviaisnorthLinkby its plain name. Any frame with no line of its own goes to every far end on that list, and that means every broadcast, so every ARP question. With one other hall, there's one place to send it.The east side is ready: Kestrel built its bridge, its tunnel card pointing at
northEdge, and plugged inmarlowEast. What's left is the flood lists.bridge fdb appendadds a far end to a list without replacing what's there, and every edge has to name every other edge.eastEdgeneedssouthEdgeadded,northEdgeneedseastEdge, and the lab does the same onsouthEdge.northEdge:~# bridge fdb show dev vxlan100 eastEdge:~# bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.0.6.2 northEdge:~# bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.0.7.2 marlowThree:~# ping -c 2 -W 1 192.168.50.11 marlowThree:~# curl -sS -m 3 192.168.50.12
Run these steps on real machines