Labs / Defensive Tools / Detect and analyse: signatures, sandbox, telemetry and decoys

Lab 3 of 5 · Detect and analyse: signatures, sandbox, telemetry and decoys

EDR telemetry: the endpoint tells on the intruder

Query a host's own live state with osquery and find a backdoor the network never saw.

Start this lab 5 steps

The idea

Hadley looks after security at Ashford & Wexford, a firm of land surveyors. Last night a contractor's account logged in to the shared workstation at 2 a.m., and the network sensor saw nothing odd afterwards. Hadley wants to check the machine from the inside before trusting that silence.

Endpoint detection and response (EDR) watches a host from the inside: the processes it runs, the ports it listens on, the files on its disk. osquery shows all of that as tables you query live with SQL, the query language of databases. Hadley uses it to find what a sensor on the network never could.

A process list, read from the inside
A process list, read from the inside
The PID ties a port to its owner
The PID ties a port to its owner
The listener that stands out against the baseline
The listener that stands out against the baseline
Two indicators, two uses
Two indicators, two uses
Nothing crossed the network
Nothing crossed the network

Steps

  1. The running machine as a table

    osquery turns the live state of a host into tables you query with SQL. The processes table is one row per running process, with its process ID (PID), name, on-disk path and full command line.

    That table is the raw material of endpoint telemetry. Instead of asking the network what a host did, you ask the host what it is doing right now.

    workstation:~# osqueryi --json "SELECT pid, name, path FROM processes ORDER BY pid"
  2. What is listening, and what owns it

    An open port is where an intruder's backdoor waits for them to come back. The listening_ports table lists every socket the host is listening on. A bare port number tells you little, though: a port only looks suspicious once you know what opened it.

    So join listening_ports to the processes table on the PID. Each listener then comes with the program that owns it, the command line that started it, and where on disk that program lives.

    Read the firm's baseline now, with an owner beside every port. A listener that shows up later then has something to stand out against.

    workstation:~# osqueryi --json "SELECT l.port, l.address, p.name, p.cmdline FROM listening_ports l JOIN processes p USING(pid) WHERE l.protocol = 6 AND l.address != '::' ORDER BY l.port"
  3. A listener nobody installed

    An intruder who gets onto a machine often leaves a process listening, so they can get back in later. That process will not announce itself, but it cannot hide from the host's own list of open ports. Run the same listening_ports query again and compare it to the baseline.

    workstation:~# osqueryi --json "SELECT l.port, l.address, p.name, p.cmdline FROM listening_ports l JOIN processes p USING(pid) WHERE l.protocol = 6 AND l.address != '::' ORDER BY l.port"
  4. Indicators you can hand over

    A listening port tells you that something is wrong. Next, gather what the team can act on.

    Two indicators of compromise matter here. The first is where the process runs from: a script in a hidden /tmp directory, where no installed program lives. The second is the hash of that file: a short fingerprint of its contents. You check the hash against threat intel, the shared lists of files already known to be malicious, and use it to find the same file on every other host.

    workstation:~# osqueryi --json "SELECT pid, name, cmdline FROM processes WHERE cmdline LIKE '%/tmp/%' AND name = 'python3'"
    workstation:~# osqueryi --json "SELECT path, sha256 FROM hash WHERE path = '/tmp/.cache/sync/updater'"
  5. Why the endpoint saw it first

    Compare this with a network sensor. A network intrusion detection system on a mirror port sees only what crosses the wire, and this backdoor has sent no traffic yet. It is only waiting.

    The host itself already knows the process, its path, its hash, its open port, and the account it runs as. That is what EDR buys Hadley: a view of what a host is, on top of what it sends.

    workstation:~# osqueryi --json "SELECT p.name, p.cmdline, u.username FROM processes p JOIN users u ON p.uid = u.uid WHERE p.cmdline LIKE '%/tmp/.cache/sync/updater%' AND p.name = 'python3'"

Run these steps on real machines

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab