Labs / Reliability & Traffic / Staying up

Lab 4 of 4 · Staying up

Server failover: the health check that said fine

Lose a server with nobody noticing, then meet the check that lies.

Start this lab 6 steps

The idea

A ferry company bought a second server so that losing one wouldn't matter. On the busiest morning of its year, it loses one.

A dead server is still on the list
A dead server is still on the list
Three failed checks, then out
Three failed checks, then out
Who answers the check
Who answers the check
One shared failure takes out every server
One shared failure takes out every server
The spare waits for DOWN
The spare waits for DOWN
Draining: no new work, old work finishes
Draining: no new work, old work finishes

Steps

  1. One server dies

    It's the morning the summer sailings go on sale, and the ferry company's booking site runs on two servers behind one address, 10.0.1.2. harbourGate is the load balancer, a machine running a program called HAProxy, and it passes each request to ticketsOne or ticketsTwo in turn. The company bought the second server so that losing one wouldn't matter.

    Nobody has tested that, so test it. From passenger, the machine playing a customer, fetch the timetable twice to see both servers answer. Then crash the app on ticketsTwo, which is what a bad update or a memory leak does on a real server, and watch what a passenger gets.

    passenger:~# curl -s http://10.0.1.2/ http://10.0.1.2/
    ticketsTwo:~# pkill python3
    passenger:~# sitewatch http://10.0.1.2/ 4
  2. Teach the balancer to look

    harbourGate keeps sending every second request to a dead server because nothing has told it otherwise. A server can't report its own crash, since the thing that would send the news is the thing that died. So the news has to come from outside: a health check is the balancer asking each server, over and over, whether it's there, and leaving out any server that stops answering.

    HAProxy's settings file has a few sections: frontend web is the door passengers come in by, backend web is the list of servers, and defaults holds settings for both. The word check on a server line turns checking on. Left at that, a check is a plain TCP connection, and a server that accepts it passes. The default-server line in the file sets the pace: inter 1s is one check a second, fall 3 is how many must fail in a row before a server is marked down, and rise 2 is how many must pass before it's let back. One failure isn't enough because one failure might be a lost packet, and pulling a healthy server out over a lost packet is how you make outages of your own.

    Add check, crash the app again, and time the gap. Then hide that gap with two more lines. retries 2 tells HAProxy to try a failed connection again, and option redispatch 1 lets every retry go to a different server. Repeating a request sounds risky on a site that takes money, so notice what's being repeated here: a connection the server refused. Nothing reached the app, so nothing can happen twice.

    passenger:~# sitewatch http://10.0.1.2/ 8
    harbourGate:~# grep DOWN /var/log/haproxy.log
    passenger:~# sitewatch http://10.0.1.2/ 8
  3. Alive, and no use

    A crash is the kind failure, because a dead program refuses connections and says so at once. The failures that hurt are quieter: a thread stuck on a lock, a disk that has stopped answering, a program waiting for something that never comes. From outside the machine looks fine.

    Freeze the app on ticketsTwo, watch the site, and ask HAProxy what it thinks of the server.

    HAProxy can also ask a harder question. option httpchk GET /ping makes each check a real HTTP request, and a server passes only by sending back a good status in time, with timeout check 1s in the file setting how long it has. Whether you need it is what the freeze tells you.

    ticketsTwo:~# pkill -STOP python3
    passenger:~# sitewatch http://10.0.1.2/ 5
    harbourGate:~# gatectl status
    passenger:~# sitewatch http://10.0.1.2/ 3
    harbourGate:~# grep DOWN /var/log/haproxy.log
  4. A deeper check

    It's the afternoon now, and seatBook, the one machine that holds every booking, falls over. All the servers lean on it, and a booking needs it, while the timetable doesn't. The new check says nothing's wrong, because /ping only asks whether the app can answer, and it can.

    Someone reads that and has the obvious idea: the check is too shallow, so make it deeper. The app already has a page for it, /health, which answers 200 only if the app can reach everything it needs, seat book included. Point the check at that, and HAProxy will know the truth.

    It sounds right, so try it. Watch /book fail, see what the balancer thinks, and fetch the timetable. Then swap the check, and when you've seen what it does, put /ping back.

    seatBook:~# pkill python3
    passenger:~# sitewatch http://10.0.1.2/book 2
    harbourGate:~# gatectl status
    passenger:~# curl -s http://10.0.1.2/
    passenger:~# curl -s http://10.0.1.2/
    passenger:~# curl -s http://10.0.1.2/
  5. A spare on the bench

    So far both servers work at once, which is called active-active. It's a good use of money, with one catch: when one dies, the other takes all of its load, so neither can run more than half full. You have to buy room for the failure you're expecting, usually one server more than you need, known as N+1.

    The other way is active-passive: a server that takes no work until the others have gone. It sounds wasteful, and people do it anyway, for machines that can't share work or a second site that's only for emergencies. The trouble with a spare is that nobody uses it, so nobody finds out it stopped working in March. That's why the health check keeps running on it.

    ticketsSpare has been sitting there since the start. The word backup on its server line makes it the spare. Add it, then take both normal servers away at once and watch the handover. Then start ticketsOne again, and see where the requests go. That return trip is called failback.

    passenger:~# curl -s http://10.0.1.2/ http://10.0.1.2/ http://10.0.1.2/ http://10.0.1.2/
    passenger:~# sitewatch http://10.0.1.2/ 8
    passenger:~# curl -s http://10.0.1.2/
    ticketsOne:~# start-tickets
    passenger:~# curl -s http://10.0.1.2/ http://10.0.1.2/
  6. Leaving on purpose

    Every server so far left by accident. Most servers leave on purpose, because somebody has an update to install, and the careless way to do it is to stop the app and let the health check find out. Anything the server was in the middle of gets cut off, and a passenger halfway through a refund gets an error and no idea whether the money moved.

    The careful way is called draining: you tell the balancer to send a server no new work. HAProxy takes that command while it runs, through gatectl, with no file to edit and no reload.

    A refund on this site takes twenty-five seconds, which is slow enough to watch. Run the first line, which starts two refunds, one on each server. You then have twenty-five seconds to drain ticketsOne and read the status.

    passenger:~# curl -s -Z --parallel-immediate http://10.0.1.2/refund http://10.0.1.2/refund
    harbourGate:~# gatectl set server web/ticketsOne state drain
    harbourGate:~# gatectl status
    passenger:~# sitewatch http://10.0.1.2/ 3
    ticketsOne:~# pkill python3
    ticketsOne:~# start-tickets
    harbourGate:~# gatectl set server web/ticketsOne state ready

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab