Labs / Observability / Metrics & alerts

Lab 1 of 1 · Metrics & alerts

Prometheus: alert on a full line before the village does

Scrape a router's traffic counters with Prometheus, turn them into megabits with PromQL, and write an alert that fires when the line fills.

Start this lab 6 steps

The idea

Every evening around eight, Thornvale's internet slows down, and Dervla hears about it from the village group chat. She volunteers for Thornvale Broadband, the co-op that runs the village's one line to the outside world. She wants the line to tell her first, so she has put Prometheus on a small box beside the co-op's Router.

Prometheus asks, the router answers
Prometheus asks, the router answers
The page holds each card's counters
The page holds each card's counters
A speed takes two readings
A speed takes two readings
`up` is about the scrape, not the router
`up` is about the scrape, not the router
A comparison is a filter
A comparison is a filter
Pending for 20 seconds, then firing
Pending for 20 seconds, then firing

Steps

  1. Ask Prometheus who answers

    Prometheus is a program that collects numbers. Every few seconds it asks each machine on its list for its current numbers, and it keeps every answer with the time it came in. Asking is called a scrape, and each machine on the list is a target.

    The machines don't send anything on their own. Prometheus goes and gets it, the way an SNMP poller asks a router for its counters. So the one doing the asking always knows when a machine has gone quiet.

    On monitor, Prometheus scrapes two targets every 5 seconds: itself, and coopRouter at 10.0.1.2, port 9100. After each scrape it writes down a few numbers of its own, and the first is up: 1 when the target answered, 0 when it didn't.

    promtool is Prometheus's command-line client. promtool query instant sends a Prometheus server a question, a query, and prints the answer as it stands right now. Ask the Prometheus on monitor, at http://localhost:9090, for up.

    monitor:~# promtool query instant http://localhost:9090 up
  2. Read the router's counters

    So what does a scrape fetch? A web page.

    On coopRouter, a small program called node_exporter reads the router's own numbers from Linux and serves them at http://10.0.1.2:9100/metrics, as plain text, one number to a line. A program that serves something's numbers like this is an exporter, and there are exporters for switches, databases, printers and nearly anything else.

    This one only reports the router's two cards: valley, facing the village's homes, and uplink, the line to the provider.

    Most of what it serves are counters. A counter only ever goes up, like a car's odometer: each card's counters start at 0 when the card is created, at boot for a real one, and add every byte and packet from then on. The other kind is a gauge, a number that goes up and down, like a temperature or the free space on a disk.

    The village has been streaming at about 20 Mbit/s since the lab started. Fetch the page from monitor, the way Prometheus does every 5 seconds.

    monitor:~# curl -s http://10.0.1.2:9100/metrics
  3. Turn a counter into megabits

    A counter alone says little. 48 million bytes since the card was created: is that a busy line or a quiet one? It depends on how long ago that was.

    What Dervla wants is a speed, and a speed takes two readings: how much the counter grew, divided by the seconds between them.

    That's what rate() does. It's part of PromQL, Prometheus's query language, and rate(node_network_receive_bytes_total{device="uplink"}[20s]) reads like this:

    • node_network_receive_bytes_total{device="uplink"}: the metric, but only the series whose device label is uplink. Braces pick series by their labels.
    • [20s]: its values from the last 20 seconds, four scrapes' worth.
    • rate(...): how fast it grew across them, per second.

    Nobody talks about a line in bytes a second, so the query goes on. * 8 turns bytes into bits, and / 1000000 turns bits into megabits.

    Run it on monitor, and see how near the village's evening comes to the line's 100.

    monitor:~# promtool query instant http://localhost:9090 'rate(node_network_receive_bytes_total{device="uplink"}[20s]) * 8 / 1000000'
  4. Block the scrape on purpose

    What happens to the numbers when the router stops answering? Find out on purpose, while you know exactly what you broke. It's the cheapest way to learn what a real outage will look like.

    A Firewall rule on coopRouter that drops packets to TCP port 9100 does it. The router keeps carrying the village's traffic and node_exporter keeps running, but Prometheus's scrapes get no answer at all. With DROP, the router throws the packets away without a word, so each scrape waits out its timeout, 4 seconds here, and gives up.

    iptables keeps a machine's packet rules in lists called chains, and INPUT is the one for packets addressed to the router itself. Block it on coopRouter and ask for up on monitor. Then delete the rule with -D, the same rule with the other letter, and ask again.

    coopRouter:~# iptables -I INPUT -p tcp --dport 9100 -j DROP
    monitor:~# promtool query instant http://localhost:9090 up
    coopRouter:~# iptables -D INPUT -p tcp --dport 9100 -j DROP
    monitor:~# promtool query instant http://localhost:9090 up
  5. Write an alerting rule

    Dervla wants a warning before the village chat lights up, and Prometheus has that built in. An alerting rule is a query Prometheus runs on its own, every 5 seconds here: its evaluation interval. Whenever the query returns anything, the alert is on.

    That works because a comparison in PromQL is a filter. ... > 80 keeps the series whose value is over 80 and drops the rest, so under 80 the query returns nothing at all. And nothing means no alert.

    A rule also gets a for:, how long the query must keep returning the series before the alert counts. That way one busy scrape doesn't wake anyone. Until then the alert is pending, and after it, firing.

    The co-op's line is 100 Mbit/s, and Dervla wants to hear at 80. Add a rule named UplinkBusy to /etc/prometheus/rules.yml on monitor, under TargetDown: the speed query from before with > 80 on the end, and for: 20s.

    The file is YAML, where indentation says what belongs to what, so the new rule's lines start in the same columns as TargetDown's. Then run the rule's query yourself.

    monitor:~# promtool query instant http://localhost:9090 'rate(node_network_receive_bytes_total{device="uplink"}[20s]) * 8 / 1000000 > 80'
  6. Fill the line and watch the alert fire

    Time for an evening peak. On homes, one more household starts a big download now, 70 Mbit/s on top of the village's 20, for up to 10 minutes. iperf3 plays the download: it's a tool for measuring how fast a network carries data, and it sends as fast as you ask.

    • iperf3 -c 10.0.2.10 -p 5202: be a client of the iperf3 server on provider, port 5202.
    • -R: reverse, so the provider sends and homes receives, like a download.
    • -b 70M: 70 megabits a second.
    • -t 600: for 600 seconds at most. It stops once you've seen the alert fire.

    It runs in the background while you watch. rate() averages over 20 seconds, so the speed climbs past 80 about 15 seconds after the start.

    Prometheus keeps its active alerts as a series too, ALERTS. Each carries the alert's name, the labels of the series that set it off, and alertstate, pending or firing. Ask for ALERTS once it's pending, and again once it fires.

    homes:~# iperf3 -c 10.0.2.10 -p 5202 -R -b 70M -t 600
    monitor:~# promtool query instant http://localhost:9090 ALERTS
    monitor:~# promtool query instant http://localhost:9090 ALERTS

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab