Labs / Troubleshooting / Where it breaks

Lab 1 of 4 · Where it breaks

Network troubleshooting: find the broken layer, one test at a time

Work a real fault like a pro: reproduce, guess, test, fix, check, write.

Start this lab 5 steps

The idea

Lunch rush at a noodle bar, and the kiosk by the door has stopped sending orders to the kitchen. Find out why with a method instead of a guess, and fix it before the queue reaches the street.

The working kiosk clears the shared parts
The working kiosk clears the shared parts
One ping cuts the layers in two
One ping cuts the layers in two
One gateway answers, the other does not exist
One gateway answers, the other does not exist
One line changed, and the next order goes through
One line changed, and the next order goes through
A theory gets tested before anything gets fixed
A theory gets tested before anything gets fixed

Steps

  1. See it fail for yourself

    The ticket reads: Door kiosk broken, orders not going through. Mia, front of house. "Broken" covers a hundred faults, so the first step of any troubleshooting method is to identify the problem, before you touch a single setting.

    That takes three things. Make it fail in front of you, so you know what broken looks like and can tell later when it's fixed. Find out how far it reaches: one machine, or every machine?

    And ask what changed, because working things rarely break on their own.

    Mia answers the last one: this morning the kiosk moved to a new stand by the door, and Sam set it up there and left a note on it. Place an order from doorKiosk to the kitchen screen at 10.0.2.20, then from counterKiosk, which sends its orders to the same kitchen screen through the same Router. Then read Sam's note.

    doorKiosk:~# curl -sS -m 5 http://10.0.2.20/order
    counterKiosk:~# curl -sS -m 5 http://10.0.2.20/order
    doorKiosk:~# cat /etc/motd
  2. Start in the middle

    Next comes a theory of probable cause: a guess that a test can confirm or kill.

    To test fast, think in layers. An order crosses every layer, from the card and cable at layer 1 to the kitchen screen's program at layer 7. You could check them one at a time, from the bottom up or the top down.

    Faster is to start in the middle. ping by address tests the bottom three layers in one go: the card, the local network and the route to the far side.

    If it works, the fault is above layer 3. If it fails, the fault is at layer 3 or below, or something filters ping itself, as many a Firewall does, which is why the error line matters.

    Each test throws away half the suspects, which is why this is called divide and conquer.

    Ping the kitchen screen from doorKiosk, by its address.

    doorKiosk:~# ping -c 3 10.0.2.20
  3. Compare with a kiosk that works

    Time to test the theory. One of the strongest tests is a comparison: put a broken machine's settings next to a working one's and look for the difference. These two kiosks sit on the same network, 10.0.1.x, and should agree on almost everything.

    The kitchen sits on a network of its own, 10.0.2.x. To reach another network, a machine hands its packets to a router, and ip route lists that choice along with the rest of its routes.

    Read the routes on both kiosks. Then run ip neigh on doorKiosk: before a machine sends anything on its own network, it asks who has the address it wants, and ip neigh lists the answers it got.

    doorKiosk:~# ip route
    counterKiosk:~# ip route
    doorKiosk:~# ip neigh
  4. Change one thing, then test

    Before you change anything, make a plan and think about what else it touches. Here the plan is short: point the door kiosk's default route, the line naming its gateway, at 10.0.1.2. It affects only this kiosk, and putting the old line back undoes it.

    The same change on a router that carries the whole shop might wait until the lunch rush is over, or go to whoever owns that router.

    Then change one thing at a time, and test after each. Change three settings at once and see the problem vanish, and you won't know which change fixed it. The other two may break something next week, and nobody will connect it to today.

    Replace the default route on doorKiosk, then place an order.

    doorKiosk:~# ip route replace default via 10.0.1.2
    doorKiosk:~# curl -sS -m 5 http://10.0.2.20/order
  5. Check the whole shop, then write it down

    The method ends with three jobs people skip: check everything, prevent a repeat, and write it down. First, check the whole system, not only the part you fixed: the counter kiosk worked all along, and it must still work.

    Then prevention. The typo still waits in the kiosk's start-up settings, so those need the same fix. And settings typed by hand will be typed wrong again one day, so the lasting fix is to let the kiosks take their settings from the router over DHCP, where nobody retypes them.

    Last, document it: what you saw, what caused it, what you changed and how you checked. Sam's note got you to the cause in minutes, and yours will do the same for the next person.

    Order from the counter kiosk. Then open Sam's note and add your line under it.

    counterKiosk:~# curl -sS -m 5 http://10.0.2.20/order

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab