Labs / Logs & SIEM / The analyst box

Lab 1 of 4 · The analyst box

Auth logs: the login that should not have worked

Read a jump host's auth log, forward it to the SIEM, and tell a guessed-in login from an engineer's own.

Start this lab 5 steps

The idea

Odile works the night shift in Thornbury Logistics' security team. Everyone who manages a server reaches it through one jump host, so every remote login in the company lands on that one box first, and sshd writes down each one, success or failure, in /var/log/auth.log.

This morning the log is longer than it should be. Somewhere in it a login worked that should never have: a password that was guessed, not known. Odile has to find that login, prove it, and tell it apart from Freya, the duty engineer, signing in to do her job.

One line, four facts
One line, four facts
Two accepted logins, two different trails
Two accepted logins, two different trails
The lines leave the host
The lines leave the host
Thirteen connections against one
Thirteen connections against one
Block one address, or remove the target
Block one address, or remove the target

Steps

  1. Every login attempt leaves a line

    A jump host is the one door into the servers: staff log in to it, then hop onward. That makes it the one place every login is recorded, so it is the first place a defender reads.

    The program that answers SSH, sshd, writes a line for each attempt to /var/log/auth.log (the path on Debian and Ubuntu; Red Hat hosts write the same lines to /var/log/secure). Two lines matter here. An attempt with the wrong password is Failed password; an attempt that got in is Accepted password. Each line carries the account, the source address, and the port, so one line tells you who tried to be whom, from where.

    Read the two kinds of line together. Everything else in the lab is about telling an ordinary login from a stolen one, and both look like an Accepted password line at first glance.

    jumphost:~# grep -E 'Failed password|Accepted password' /var/log/auth.log
  2. A wall of failures, then one success

    Guessing a password online means trying one after another against the live service. Every wrong guess is a Failed password line; the moment one works, the next line is Accepted password, from the same source for the same account. That shape, many failures then a success from one host, is the signature of a password guessed in, and it is what MITRE ATT&CK files under Brute Force.

    A normal login has no such run in front of it. Freya typed her password right the first time, so her Accepted line stands alone. A stray failure or two before a success is usually someone mistyping their own password; what marks a break-in is the volume from one source (a dozen here), all aimed at the one account.

    Count the failures aimed at deploy from that one address. grep -c prints how many lines match instead of the lines themselves.

    jumphost:~# grep -c 'Failed password for deploy from 10.0.1.50' /var/log/auth.log
  3. Forward the log to the SIEM

    Reading auth.log on one box worked because you knew which box. A real company has hundreds, and an intruder who lands on one copies its log or wipes it. So hosts forward their logs to a SIEM (security information and event management): one place that holds every machine's records and answers questions across all of them at once. That way the evidence outlives the host, and a search covers every machine.

    Forward this jump host's auth log to the analyst box, then ask the SIEM the same question you asked the file a moment ago. logger sends log lines over the network with syslog, the standard way a machine hands log lines to a collector: --udp sends each line as a UDP datagram to --server and --port (514 is syslog's port), --rfc3164 writes them in the classic syslog format the collector expects, --file reads the lines from a file instead of waiting for you to type them, and --tag sshd labels them so they stand out later. UDP is fire-and-forget. It is fast, but a datagram can be dropped, so for evidence that must not go missing a real site would forward over TCP or TLS.

    The SIEM answers in PPL (Piped Processing Language), a query language that reads left to right like a pipeline: start at an index (syslog, where forwarded host logs land), filter, then count. The filter here, like(message, "%Failed password%"), keeps the lines whose message holds that text. The % on each side matches any characters around it.

    jumphost:~# logger --udp --server 10.0.1.20 --port 514 --tag sshd --rfc3164 --file /var/log/auth.log
    analyst:~# curl -s 'localhost:9200/_plugins/_ppl?format=csv' -H 'content-type: application/json' -d '{"query":"source=syslog | where like(message, \"%Failed password%\") | stats count()"}'
  4. Find the guessing host by its connections

    The auth log names the account, but its source address sits in free text, hard to total up. The network sensor is tidier. Zeek watches a mirrored copy of the office traffic and writes one line per connection to its conn.log, each with clean fields: who connected (id.orig_h), to whom, and to which port (id.resp_p). SSH is port 22.

    A guessing run is loud on the wire. In this run each guess opened its own SSH connection, so a dozen guesses are a dozen connections to port 22 from one host, and the login that worked is one more. One connection can in principle carry a few attempts: sshd allows up to MaxAuthTries, six by default. So a connection count is a lower bound on attempts; the auth log, one line per attempt, is the exact tally. Here the two line up.

    A real user makes a single connection. Ask the SIEM to count SSH connections per source, busiest first. The leading - in sort - attempts means descending. It is the same question the auth log answered, now from a second, independent log.

    analyst:~# curl -s 'localhost:9200/_plugins/_ppl?format=csv' -H 'content-type: application/json' -d '{"query":"source=zeek-conn | where id.resp_p = 22 | stats count() as attempts by id.orig_h | sort - attempts"}'
  5. The verdict, and shutting the door

    Put it together. deploy was guessed in from 10.0.1.50: twelve failures then a success, and the same host connecting thirteen times. freya from 10.0.1.30 is the duty engineer: one login, no failures. One line is a break-in; the other is routine work.

    Finding the break-in is half the job; closing the door is the other half. Blocking 10.0.1.50 stops this host, but the attacker can move to another address and the weak password is still there to guess. The fix that lasts is to take passwords off the door: with SSH keys only, there is nothing to guess. Read back the one line that is the moment of the break-in.

    This was one host guessing one account, the loudest shape. A single common password tried across many accounts, or guesses spread over many addresses at once, slip past a per-source count; correlating across accounts and sources comes later on this road. And a confirmed success is where the response starts, not ends: next you would hunt what deploy did once it was inside.

    jumphost:~# grep 'Accepted password for deploy from 10.0.1.50' /var/log/auth.log

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab