Commands / Services and tunnels

Services and tunnels

jq

Also json, json query, json filter, rest api, pretty print json, eve.json, JSON lines

Reads JSON: pretty-prints it, picks fields out of it and filters arrays, as a REST API's answers need.

A REST API answers in JSON, which is hard to read in one line and harder to use in a script. jq . prints it indented; a path picks a field (.token, .routes[0].via); .[] walks an array, and select(…) keeps the items that match. -r prints a string without its quotes, so the result can go straight into the next command (a token into a header). It reads a file of one JSON object per line (JSON lines, like Suricata's eve.json) one object at a time, so select(.event_type=="alert") keeps the alerts of a log.

jq command usage

jq [-r] FILTER [FILE]
curl -s URL | jq FILTER

Options

.
the whole document, indented
.field / .a.b / .[0]
one field, a nested field, one item of an array
.[] | select(.x == "y")
each item of an array, keeping those that match
-r
raw: print strings without quotes (for use in a script)
-c
compact: each result on one line
length, keys
how many items, which field names

jq examples

curl -s -u admin:netlab http://10.0.1.10:8080/api/devices | jq .
Prints the API's answer indented and coloured.
curl -s -u admin:netlab http://10.0.1.10:8080/api/routes | jq -r '.[] | select(.device == "edgeRouter") | .via'
Prints the next hop of every route the API holds for edgeRouter.
jq -r .token token.json
Prints the token field without quotes, ready for an Authorization header.
jq -c 'select(.event_type=="alert") | {src_ip, signature: .alert.signature}' /var/log/suricata/eve.json
Lists each Suricata alert's source address and signature, one per line.
tail -n 6 /var/log/caddy-access.json | jq -r '.request.proto + " " + .request.uri'
Prints each logged request's HTTP version and path as plain text.
jq '.name' device.json
Prints one field of the object.
jq '.[0]' devices.json
Prints the first element of an array.
jq -r '.[].name' devices.json
Prints the name of every element, one per line, without quotes.
jq 'length' devices.json
Counts an array's elements (or an object's keys).
jq 'keys' device.json
Lists the object's keys, sorted: what you can ask for.
jq '[.[] | select(.state == "up")] | length' interfaces.json
Counts the elements whose state is up.
jq -r '.[] | [.name, .address] | @tsv' devices.json
Prints a table, one line per element and a tab between the fields; @csv gives CSV.
jq --arg dev edgeRouter '.[] | select(.device == $dev)' routes.json
Passes a shell value in as $dev instead of pasting it inside the filter.
jq 'sort_by(.rtt) | reverse | .[:3]' probes.json
Sorts by rtt and keeps the three slowest.
jq '.mtu = 9000 | del(.password)' device.json
Prints the object with one field changed and one removed; the file itself stays as it was.
jq -s 'map(.bytes) | add' flows.jsonl
Reads JSON lines into one array (-s) and adds up their bytes.
ip -j addr | jq -r '.[] | .ifname + " " + (.addr_info[0].local // "-")'
Lists each card and its first address from ip's JSON output, a dash where it has none.

Reading the output

$ tail -n 1 /var/log/caddy-access.json | jq -c '{proto: .request.proto, uri: .request.uri, status}'
{"proto":"HTTP/3.0","uri":"/","status":200}
  • Each input object gives one result; -c keeps it on one line.

Pitfalls

  • parse error: the input is not JSON (an HTML error page, or text before the JSON).
  • Without -r a string keeps its quotes, and a header built from it is wrong.
  • A missing field is null, not an error: check the spelling when jq prints null.

Run it on real machines

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Start with a free account: Google or email, no card.

Open this lab