Labs / Reliability & Traffic / Staying up

Lab 2 of 4 ยท Staying up

VRRP failover: two routers, one gateway address

Pull the plug on a gateway and watch a spare router take its address.

Start this lab 5 steps

The idea

A radio station can't go quiet, and everything in its studio reaches the transmitter through one Router. Buy a spare, then make the studio's machine use it without ever being told.

A MAC that belongs to the group
A MAC that belongs to the group
Only the master speaks
Only the master speaks
The address moves, and nothing else does
The address moves, and nothing else does
The higher priority takes the job back
The higher priority takes the job back
Each side picks its own master
Each side picks its own master

Steps

  1. One gateway address, two routers

    The studio's audio reaches the transmitter through mainRouter, and if that router dies the station plays dead air. Buying a second router is the cheap half. The hard half is that every machine has one default gateway in its settings, and when that router stops answering, the machine doesn't go looking for another.

    First-hop redundancy turns the problem around: the machines keep one gateway address, and that address stops belonging to any one box.

    Start on studioPc. Read its gateway, ask who's at that address, and trace the way to the transmitter.

    studioPc:~# ip route
    studioPc:~# ip neigh show 10.0.1.254
    studioPc:~# traceroute -n 10.0.2.20
  2. Listen to the routers agree

    For this to work the two routers have to stay in step, and they do it over the network with VRRP, the Virtual Router Redundancy Protocol. On these routers a Linux program called keepalived runs it.

    Cisco got there first with HSRP in the 1990s, but that was theirs alone, so the IETF, the body that writes internet standards, wrote VRRP as the open version every vendor could speak. The idea is plain enough: routers in a group have a priority, and the highest becomes master and holds the address. Here mainRouter has priority 150, spareRouter 100.

    Check who holds the address, on a virtual card called vrrp.1 that keepalived made for this group. Then listen on studioPc: VRRP messages go to a Multicast address, one that any machine on the network can hear.

    mainRouter:~# ip -br addr show vrrp.1
    spareRouter:~# ip -br addr show vrrp.1
    studioPc:~# tcpdump -n -v -c 3 vrrp
  3. Cut the power

    It's time to break it. In a moment mainRouter loses power: no goodbye, no warning, cards dark. studioPc is sending a ping five times a second through the gateway, the lab's stand-in for the audio stream, and the meter calls each one a probe.

    spareRouter can't tell a dead master from one late packet, so it doesn't move at the first silence. It counts missed advertisements, and adds a small extra wait worked out from its priority. With several backups, the best one moves first and the others hear it and stand down.

    studioPc:~# ping -i 0.2 -c 100 -W 1 10.0.2.20
    studioPc:~# ip neigh show 10.0.1.254
    studioPc:~# traceroute -n 10.0.2.20
  4. The main router comes back

    Someone finds the breaker, and mainRouter boots. spareRouter is working fine right now, and traffic is flowing.

    There are two schools of thought here. If the main router is the better box, on the better line, you want it back in charge. But a router fresh from a boot may not be ready yet, and one that's failing may go down again a minute later.

    Run the meter again while the power comes back.

    studioPc:~# ping -i 0.2 -c 100 -W 1 10.0.2.20
    studioPc:~# traceroute -n 10.0.2.20
    studioPc:~# tcpdump -n -v -c 3 vrrp
  5. Up, and of no use

    A digger goes through the cable between mainRouter and the tower. The router is fine, and so is its power.

    Run the same checks as before, and listen on both sides of the network, since each network's group runs on its own.

    keepalived has a setting for the day those checks look bad: track_interface names a card, and a group gives up its job when that card goes down.

    studioPc:~# ping -c 3 -W 1 10.0.2.20
    studioPc:~# tcpdump -n -v -c 2 vrrp
    transmitter:~# tcpdump -n -v -c 2 vrrp
    studioPc:~# ping -c 3 -W 1 10.0.2.20
    studioPc:~# traceroute -n 10.0.2.20

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab