The idea
Rafael has an authorised engagement against Zephyr Air, a regional airline, starting with its public booking server at 10.0.1.20. His whole first hour is reconnaissance: learn what the target is before touching how it breaks.
He opens no browser. A browser renders the page and throws the rest away; Rafael wants the rest. With curl alone he reads what the server volunteers to any client: the software and version in its headers, where it redirects, the paths it asks crawlers to skip, and a file someone left where it should not be.
Steps
-
Ask the server to introduce itself
Every HTTP reply comes in two parts: a block of headers, then the body (the page). The first line of the headers is the status line, a number that says how the request went. A browser shows you the body. The headers are where the server describes itself, and they are the first thing a tester reads.
curl -Ifetches the headers only. It sends aHEADrequest, which means "send me the headers you would send for aGET, but not the body". So you read the server's own description of itself without pulling a single page. Point it at the site root and read what comes back.laptop:~# curl -sI http://10.0.1.20/
-
Follow where it sends you
The root answered
301 Moved Permanentlywith aLocationheader. A redirect is the server saying "not here, there", and the real application lives at there. A browser follows it without a word, so a user never notices; a tester has to follow it on purpose, because the thing worth mapping is wherever the chain ends.curldoes not follow redirects unless you tell it to.-Lmakes it follow theLocationheader to the next response, and the next, until a page actually answers. Keep-Iso you watch each hop as headers.laptop:~# curl -sIL http://10.0.1.20/
-
Read what they ask crawlers to skip
Almost every site serves a file at
/robots.txt. It is a politeness note to search engines: a list of paths the owner would rather was not crawled and indexed. Google obeys it; an attacker reads it as a map.Think about what ends up in that list. An admin does not ask Google to skip the public home page; they ask it to skip the pages they would rather the world did not find: control panels, staging areas, backups. The file that is meant to hide those paths from a search index is the fastest way to learn they exist.
laptop:~# curl -s http://10.0.1.20/robots.txt
-
The file left in the open
Nothing on the site links to
/admin-dashboard/or/backups/, butrobots.txtnamed both. A path no page links to is not hidden, only unlinked: the server answers a request for it like any other.Backups under the web root are a classic, costly slip: an editor's save file, a
config.baka deploy script copied and forgot. The web server has no idea a file is sensitive; it serves it like any other.Start with the staff page, then the backups folder itself. A folder that will not list its files can still serve any file you name, and backup names are predictable:
config.bak,backup.zipanddb.sqlsit near the top of every Content discovery wordlist.laptop:~# curl -s http://10.0.1.20/admin-dashboard/ laptop:~# curl -sI http://10.0.1.20/backups/ laptop:~# curl -s http://10.0.1.20/backups/config.bak
-
Shut the server up
Now swap seats. You run Zephyr Air's servers, and Rafael's report lists four gifts: the version banner, the framework header, a
robots.txtthat advertises sensitive paths, and a backup under the web root. The fix is to stop volunteering what recon reads.Turn off
server_tokensso nginx stops printing its version, drop theX-Powered-Byheader, and get the backup off the web server entirely. A backup never belongs under the web root. Then look at the server's own access log and ask the harder question: could you even have caught Rafael doing this?web:~# nginx -s reload web:~# rm -f /srv/www/backups/config.bak laptop:~# curl -sI http://10.0.1.20/ laptop:~# curl -s http://10.0.1.20/backups/config.bak web:~# tail -n 20 /var/log/nginx/access.log
Run these steps on real machines
Keep reading
- OWASP WSTG: Fingerprint Web Server article
- MDN: the HTTP HEAD method article
- Google: how to write and submit a robots.txt file article
- Everything curl: HTTP responses article