Labs / Troubleshooting / Reading the counters

Lab 1 of 1 · Reading the counters

Cable and interface errors: read a port's state and counters, and find the bad link

Read port states and counters to find a switched-off port, bad cable or MTU clash.

Start this lab 4 steps

The idea

Fern & Pine, an architecture studio, moved into a new office over the weekend: two desks and a file server, one cable each into one Switch. All week something goes wrong in the physical part of the network: the ports and the cables. Learn to read what a port says about itself, and the numbers it keeps.

A port that is off has no link either
A port that is off has no link either
A port counts from its own side
A port counts from its own side
Damage is counted where it arrives
Damage is counted where it arrives
The port drops what is bigger than its MTU
The port drops what is bigger than its MTU

Steps

  1. Switched off, or no link

    Monday. editDesk can't reach anything, and the installer who set up studioSwitch over the weekend has gone home.

    Start with the physical part: the ports and the cables. On studioSwitch, the switch itself is called br0. Each of its ports is named after the device at the other end of its cable, so the port called editDesk is the one editDesk plugs into.

    A switch port, like any network card, has two states that people mix up all the time. Its settings switch it on or off. Separately, it has a live link or it doesn't: a working cable to a working card at the other end.

    ip link shows both, as flags between < >. UP means switched on, and LOWER_UP means the link is live. A port that's on with no link shows NO-CARRIER in place of LOWER_UP.

    editDesk:~# ping -c 3 -W 1 10.0.1.20
    studioSwitch:~# ip -br link show master br0
    studioSwitch:~# ip link set editDesk up
    editDesk:~# ping -c 3 -W 1 10.0.1.20
  2. Counters only count up

    Tuesday, and nothing is broken. That makes it the day to learn what normal looks like, before you need it.

    Every card and every switch port keeps counters: frames and bytes received (RX) and sent (TX), and the ones it dropped or found damaged. They start at zero when the card starts and only ever go up.

    So one reading tells you little. RX errors 312 could be a broken cable, or one bad minute three weeks ago. Read the counters, let something happen, read them again: the difference is what happened in between.

    RX and TX are the port's own point of view. On the switch's port to fileServer, RX is what the switch received from the server, and TX what it sent to it.

    Read them while designDesk downloads the studio's 2 MB drawing set.

    studioSwitch:~# ip -s link show fileServer
    designDesk:~# curl -sS -o /dev/null -m 5 -w "%{size_download} bytes\n" http://10.0.1.20/plans.pdf
    studioSwitch:~# ip -s link show fileServer
  3. A cable that damages what it carries

    Wednesday. The designers say the file server feels sluggish, and pings to it go missing. Last night the office manager rolled a chair over a cable, and isn't sure which.

    On real hardware, every Ethernet frame ends with a check value, computed from every bit in it. The card at the receiving end works it out again. When the two differ, a bit changed on the way: the card throws the frame away and adds one to its CRC errors.

    So a damaged cable shows up twice: as packet loss, and as CRC errors on the card at the receiving end of that cable.

    This lab's cables are virtual and carry no check value, so a card here can't spot the damage itself. It's caught further up instead: IP, ICMP and TCP carry checksums of their own, and nstat counts the packets that fail them.

    First find which cable. Ping the server, then the other desk, from designDesk.

    designDesk:~# ping -c 20 -i 0.2 -s 1000 10.0.1.20
    designDesk:~# ping -c 20 -i 0.2 -s 1000 10.0.1.11
    fileServer:~# ip -s -s link show eth0
    fileServer:~# nstat -az IcmpInCsumErrors TcpInCsumErrors
    designDesk:~# ping -c 20 -i 0.2 -s 1000 10.0.1.20
    fileServer:~# nstat -az IcmpInCsumErrors TcpInCsumErrors
  4. Too big for the port

    Thursday. Since a contractor's visit last night, designDesk can open the studio's file list, but plans.pdf never arrives.

    editDesk downloads the same file in a blink.

    Every port has an MTU: the biggest packet it carries in one frame, 1500 bytes on ordinary Ethernet. A frame that arrives bigger than a port allows isn't split: a switch can't split frames, and can't tell the sender either. The port drops it, and Cisco counts it as a giant.

    Small things fit, so they work: a file list, a ping, a short reply. A download fills every packet to the brim, so it's the thing that breaks. When small works and big stalls, think MTU.

    To measure it, send pings of known sizes. -M do forbids splitting a ping into pieces on the way, and -s sets its size: 1472 bytes of data plus 28 of headers makes a full 1500.

    designDesk:~# curl -sS -m 5 http://10.0.1.20/
    designDesk:~# curl -sS -o /dev/null -m 5 -w "%{size_download} bytes\n" http://10.0.1.20/plans.pdf
    studioSwitch:~# ip -s link show designDesk
    designDesk:~# ping -c 2 -W 1 -M do -s 1372 10.0.1.20
    designDesk:~# ping -c 2 -W 1 -M do -s 1472 10.0.1.20
    studioSwitch:~# ip -s link show designDesk
    studioSwitch:~# ip link show master br0
    studioSwitch:~# ip link set designDesk mtu 1500
    designDesk:~# curl -sS -o /dev/null -m 5 -w "%{size_download} bytes\n" http://10.0.1.20/plans.pdf

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab