Labs / Observability / Logs, counters & flows

Lab 3 of 3 · Logs, counters & flows

NetFlow: find who fills the library's line

Export flow records from a router to a collector, rank who talks the most, and catch a laptop scanning the internet.

Start this lab 5 steps

The idea

Every afternoon the patrons' internet at Brightwater Public Library slows to a crawl, and Marisol, who looks after the library's computers two days a week, has to find out why. The library promises patrons that nobody reads what they do online, so capturing their traffic is out. Flow records are the way in: the Router writes down who talked to whom and how much, and never what was said.

One conversation, two records
One conversation, two records
A long flow is cut every 30 seconds
A long flow is cut every 30 seconds
The router never learns they were lost
The router never learns they were lost
One download fills both top rows
One download fills both top rows
Flows and bytes rank differently
Flows and bytes rank differently

Steps

  1. Fetch a page and find its flow

    A flow is one conversation, seen from outside. libraryRouter doesn't keep a word of what's said. It keeps the envelope: who talked to whom, on which ports, over which protocol, how many packets and bytes, and when it started and stopped.

    Five of those fields make the flow's key, the 5-tuple: both addresses, both ports and the protocol. Every packet with the same five adds to the same record, so a million packets of one download make a single line.

    The router runs an exporter, softflowd, which keeps a record for each conversation crossing its patrons card. When a conversation ends, the record travels to flowCollector on the office network, where a collector, nfcapd, writes it to a file under /var/flows. nfdump reads the files back.

    The whole arrangement is NetFlow, Cisco's name, which stuck for flow records from any vendor.

    203.0.113.10 is a game company's download server, a mirror, out on the internet. Fetch its front page from laptop1. Then ask the collector for every flow involving laptop1's address, 10.0.1.21.

    laptop1:~# curl -s http://203.0.113.10/
    flowCollector:~# nfdump -R /var/flows 'host 10.0.1.21'
  2. Watch a long download arrive in slices

    laptop2 has been pulling a 2 GB game patch from the mirror since the lab started. At 20 megabits a second that takes about 14 minutes. So when does the router report a conversation that won't stop?

    An exporter closes a record for one of three reasons:

    • The conversation ends: TCP says goodbye, or one end resets it.
    • It goes quiet for longer than the inactive timeout.
    • It has run for the active timeout. The router sends what it has so far and starts a fresh record for the same 5-tuple.

    Without that last rule, a 3-hour backup would reach the collector 3 hours late, all at once.

    Real routers use an inactive timeout of about 15 seconds and an active timeout of 1 to 30 minutes. libraryRouter uses 10 seconds and 30 seconds, so a lab session sees several slices.

    Records still open live in the router's flow cache until one of those three cuts them off. List the download's records that have reached the collector.

    flowCollector:~# nfdump -R /var/flows -c 4 'host 10.0.1.22 and bytes > 100000'
  3. Switch the collector off

    The records travel as ordinary packets, over UDP to port 2055 on flowCollector, each packet carrying a batch of them. That path is where flow monitoring breaks most often.

    UDP has no acknowledgements. The router sends each report once and never hears whether anyone took it. When a UDP packet reaches a machine where no program has its port open, the machine says so with an ICMP message.

    You're about to break this on purpose. On flowCollector, stop nfcapd, the program that writes the files, and capture what still arrives on its office card. Then start nfcapd again, the way the lab started it.

    flowCollector:~# pkill -x nfcapd
    flowCollector:~# tcpdump -n -l -c 4 -i office 'udp port 2055 or icmp'
    flowCollector:~# nfcapd -w /var/flows -p 2055 -t 10 -D
  4. Rank the talkers by bytes

    Now Marisol's real question: who is filling the line?

    nfdump -s builds a top-N table instead of listing records. It adds up the records by one field and sorts the totals:

    • -s ip/bytes groups by address, counting a record under both its source and its destination, and sorts by bytes.
    • -s srcip/bytes counts only the sender, which tells uploads from downloads.
    • -n 5 keeps the top five.

    The table shows flows, packets and bytes for each row, then three averages: pps packets a second, bps bits a second and bpp bytes a packet. Its percentages are shares of everything in the files read. Every flow tool, from a free script to a big commercial dashboard, is mostly this query with a graph around it.

    Bytes answer "who uses the line". Flows answer "who starts the most conversations". The two lists are rarely the same, so keep both in mind.

    Rank the addresses by bytes, then rank only the senders.

    flowCollector:~# nfdump -R /var/flows -s ip/bytes -n 5
    flowCollector:~# nfdump -R /var/flows -s srcip/bytes -n 3
  5. Count conversations, not bytes

    Bytes find the heavy users. Counting flows finds something else: machines that start an odd number of conversations, most of them tiny.

    A flow of a packet or two, 40 bytes each, to TCP port 23 is a bare SYN, the first step of a TCP handshake, that went no further. One of those is a typo, and hundreds, each to a different address, are a port scan: a machine knocking on door after door to see which opens.

    Port 23 is telnet, which infected devices scan for to find more with factory passwords.

    Every record also keeps the TCP flags seen in the flow, all together, and -o extended prints them in 8 fixed places, with a dot for each flag never seen. S is SYN, A is ACK, R is reset, F is FIN and P is push. A finished conversation shows ...AP.SF, and a sender that never got past its SYN shows ......S..

    Rank the senders by flows, then read the flags on the busiest one's records.

    flowCollector:~# nfdump -R /var/flows -s srcip/flows -n 5
    flowCollector:~# nfdump -R /var/flows -o extended -c 6 'src host 10.0.1.23'

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab