Labs / Protocols / The web & remote shells

Lab 1 of 4 · The web & remote shells

HTTP: read the requests behind every web page

Read HTTP requests line by line, then find the order a redirect lost.

Start this lab 7 steps

The idea

Noor's bakery shares a web server with a boatyard. Her browser opens the bakery every time, but the small order script she wrote gets the boatyard's page, and once she fixes that, orders vanish with no error at all.

A request and a reply have the same three parts
A request and a reply have the same three parts
The address finds the machine, the Host line finds the site
The address finds the machine, the Host line finds the site
nginx wrote all three replies
nginx wrote all three replies
A POST puts its data after the empty line
A POST puts its data after the empty line
One digit decides if the order survives
One digit decides if the order survives
The cookie is a number, the basket stays on the server
The cookie is a number, the basket stays on the server
Inside the hour no question, after it a cheap one
Inside the hour no question, after it a cheap one

Steps

  1. Read one request and its reply

    A browser hides a conversation behind every page. Underneath, it sends a text message to a server and gets a text message back, over a TCP connection, usually on port 80 or 443. The rules for those two messages are HTTP, the Hypertext Transfer Protocol.

    Tim Berners-Lee wrote the first version around 1990 so that researchers could fetch each other's documents, and up to version 1.1 it stayed plain text anyone could read. Later versions pack the same lines into binary, and tools print them back in this shape. That helps you, because a problem on a website usually shows up in these lines.

    curl is a command-line web client: it asks for a page the way a browser does, and prints what it gets. With -v it prints the conversation too, and -s hides a progress meter that would get in the way. Lines that start with > went out, lines that start with < came back, and lines that start with * are curl's own notes.

    laptop:~# curl -sv http://bakery.lab/
  2. Host: one address, many sites

    Noor's script doesn't use the name. She wrote the server's address into it, 10.0.2.80, to skip the name lookup. Both bakery.lab and boatyard.lab point at that address, a machine called sharedHost, and they share one server program on port 80.

    Think about what the server knows when a connection arrives. DNS turned the name into an address and stopped there, and a TCP connection carries addresses and ports, with no names. So the only place a name can travel is inside the request, in the Host header.

    That header exists for exactly this reason. Early web servers ran one site per address, and the header let a single address hold many sites, which is called virtual hosting. From version 1.1 of HTTP every request must carry it, and today a small site often shares its address with strangers.

    First run Noor's script's request. Then type a request by hand: nc opens a plain TCP connection and sends what you type, so the Host line is yours to write.

    laptop:~# curl -si http://10.0.2.80/
    laptop:~# nc bakery.lab 80
    sharedHost:~# grep -n -e server_name -e listen /etc/nginx/nginx.conf
  3. Status codes: what the first digit tells you

    Noor's script now uses the name, and it asks for the menu at /menu. A person reading a browser sees a page or an error page. A script sees neither: it gets a three-digit status code, and it has to read it.

    The designers gave the first digit a meaning, so even a code you've never seen tells you what to do:

    • 2xx: it worked.
    • 3xx: look somewhere else.
    • 4xx: the server thinks the request is at fault.
    • 5xx: the server failed.

    The first digit tells you whose problem it is. A 4xx usually fails the same way until the request changes, and a 5xx may pass on a retry, because the request was fine. That holds for fetching a page: repeating an order is riskier, since the first try may have gone through.

    The standard also has 1xx, a note sent while the real reply is on its way, and the WebSockets lab uses one.

    Try three requests. In the third one, the program behind the order page has been stopped, the way it would stop in a crash.

    laptop:~# curl -siL http://bakery.lab/menu
    laptop:~# curl -si http://bakery.lab/cakes
    laptop:~# curl -si http://bakery.lab/order/
  4. POST: hand data to the server

    GET asks for something. Noor's script has to deliver something, an order, and that takes another method.

    The method is the first word of the request, and it tells the server what you want done. POST hands over data for the server to act on.

    PUT replaces a thing, DELETE removes it, and HEAD asks for the headers of a GET without its body.

    A method is a meaning both sides agree on, and nothing enforces it. A server could delete something on GET. People who build sites take care that it doesn't, because browsers and search engines repeat GET requests freely.

    A GET can carry small details in the address, as in /search?q=rye, and those end up in logs and browser history. A POST carries its data in the body, the part after the empty line. So a request now has the same three parts as a reply, and the headers have to describe the body: Content-Type says what kind of data it is, and Content-Length says how many bytes.

    curl -d sends a POST with the data you give it. The order program on /order/ takes a form field called item.

    laptop:~# curl -sv -d item=rye http://bakery.lab/order/
    laptop:~# curl -si -X DELETE http://bakery.lab/order/
  5. Redirects: how an order gets lost

    Noor's script worked once, then a customer's order disappeared. Her script posts to /order, with no slash, and she added -L so that it follows redirects, as a good script should. She sees 200 OK every time.

    You've seen a redirect work on a GET. A client that follows one sends a second request to the new address. What should that second request be when the first was a POST?

    The first HTTP specification said to keep the method and ask the user first. Browsers of the time didn't: they turned the POST into a GET and dropped the body, and sites came to depend on that.

    The later standards gave up on the question. To this day, a client that follows a 301 or 302 is allowed to turn a POST into a GET and drop the body.

    Two newer codes say what they mean: 307 and 308 keep the method and the body. 308 only became a standard in 2015, so plenty of servers and programs still use 301 from habit.

    Send the order the way Noor's script does, and read the server's log. Then add a rule to the settings that stops the loss.

    laptop:~# curl -siL -d item=rye http://bakery.lab/order
    sharedHost:~# tail -n 2 /var/log/nginx/access.log
  6. Cookies: how a server remembers you

    Noor's orders arrive now, one at a time. Her script adds rye, then adds seeded, and the basket shows only seeded.

    The cause is in how HTTP was designed. Each request stands alone, and nothing in the protocol links it to the one before, even on the same connection.

    This is called being stateless, and it keeps servers simple: a server can answer anyone at any moment, with no memory of them. It makes a shopping basket impossible, though.

    In 1994 Lou Montulli, at Netscape, solved this with a small addition. The server sends a short piece of text in a Set-Cookie header, the browser stores it, and the browser sends it back in a Cookie header on every later request to that site.

    An HTTP cookie is that piece of text. HTTP stays stateless, and the client carries the memory.

    curl keeps no cookies unless you ask. -c jar.txt writes them to a file, and -b jar.txt sends them back.

    laptop:~# curl -si -c jar.txt -d item=rye http://bakery.lab/order/
    laptop:~# curl -s http://bakery.lab/order/
    laptop:~# curl -sv -b jar.txt http://bakery.lab/order/
  7. Caching: ask whether it changed

    A bakery's menu changes rarely, and a customer's browser may ask for it a hundred times. Fetching it each time wastes bandwidth and makes the page slower, and the fix is an HTTP cache: the client keeps a copy and reuses it.

    The hard part is knowing when the copy has gone out of date, and HTTP gives the server two tools. An expiry time says how long to reuse the copy without asking, and a tag makes the question afterwards cheap.

    The expiry time is Cache-Control: max-age=3600. It says the copy is good for an hour, and during that hour a browser doesn't even ask.

    The tag is an ETag, a short label the server puts on each version of a file, and it changes when the file does. When the hour is up, the browser sends the tag back in an If-None-Match header, which asks "has it changed from this one?"

    curl keeps no cache of its own, but it can do that check by hand: --etag-save writes the tag to a file, and --etag-compare sends it back.

    laptop:~# curl -si --etag-save tag.txt http://bakery.lab/menu/
    laptop:~# curl -si --etag-compare tag.txt http://bakery.lab/menu/

Run these steps on real machines

Keep reading

Learn it on real machines.

Every lab is a small network of real Linux machines in your browser. Sign in with Google or email to start.

Start this lab