MTU

Counts the IP header, the transport header and the data, not the Ethernet header. Tunnels and some lines have less (1400 to 1492) because they add headers of their own. A packet larger than the next link's MTU is fragmented, or refused with an ICMP message when it says Don't Fragment.

How MTU works, step by step

The MTU is a property of each link, set per interface. On Ethernet it is 1500 bytes of IP packet (RFC 894). The Ethernet header (14 bytes) and frame check (4) come on top, so a full frame on the wire is 1518 bytes, or 1522 with a VLAN tag. Follow a laptop at 10.0.1.10 sending to a file server at 10.0.3.10, where the path crosses a 1400-byte link between officeRouter and branchRouter:

1. The sender checks its own card first. A packet larger than the outgoing interface's MTU is never sent whole: the kernel fragments it or refuses it with message too long.
2. TCP picks its segment size from the MTU. In the SYN each side announces an MSS of its own MTU minus 40 (20 IP + 20 TCP): 1460 on Ethernet. Neither side knows about the 1400-byte link in the middle.
3. Each router checks the next link. A 1500-byte packet reaches officeRouter, whose next link carries 1400. Without Don't Fragment (DF) it is cut into fragments. With DF set, as on almost all TCP traffic, it is dropped and the router sends ICMP type 3 code 4, fragmentation needed, with the MTU 1400 in the message (RFC 1191).
4. The sender lowers its size for that destination and caches the path MTU. That is path MTU discovery.

Tunnels are the usual reason a link has less than 1500: they spend part of the outer packet on headers. GRE costs 24 bytes (1476), PPPoE 8 (1492), VXLAN 50 (1450), WireGuard 60 over IPv4 or 80 over IPv6 (Linux defaults to 1420). The limits run the other way too: IPv4 requires every link to carry 68 bytes, IPv6 1280, and data centres raise the underlay to 9000 or 9216 (jumbo frames) so tenants keep 1500.

What it looks like on a Linux machine

In the MTU and fragmentation lab the router's two cards show their MTUs in ip addr show, mtu 1500 on the office side and mtu 1400 on the tunnel side. The largest ping that fits a 1500-byte card carries 1472 bytes of data (1472 + 8 ICMP + 20 IP); one byte more is refused before it leaves:

$ ping -c1 -M do -s 1473 10.0.1.2
ping: local error: message too long, mtu=1500

Across the narrow link, the first router answers and the laptop remembers:

$ ping -c2 -M do -s 1450 10.0.3.10
PING 10.0.3.10 (10.0.3.10) 1450(1478) bytes of data.
From 10.0.1.2 icmp_seq=1 Frag needed and DF set (mtu = 1400)
ping: local error: message too long, mtu=1400
$ ip route get 10.0.3.10
10.0.3.10 via 10.0.1.2 dev eth0 src 10.0.1.10 uid 0
    cache expires 597sec mtu 1400

The learned value lives about ten minutes (net.ipv4.route.mtu_expires, 600 s). ip link set dev eth0 mtu 1400 changes a card's MTU. In the GRE tunnels lab the tunnel card comes up at mtu 1476 on its own, and in the VXLAN lab a full-size ping fails with mtu=1450.

What goes wrong, and how you see it

Small works, big hangs. A firewall drops the fragmentation-needed messages, so the server keeps sending 1500-byte packets into a 1400-byte link. Pages half-load, SSH freezes on long output, curl ends with fewer bytes than the file. That is a PMTU black hole; MSS clamping avoids it for TCP: iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu.

Two ends that disagree. One card is set to 1400 and its neighbour to 1500. The 1500 side sends full frames, the other drops them on arrival. In the interface errors lab pings up to -s 1372 answer and bigger ones get 0 received; real NICs count these as giants (rx_length_errors), a veth as rx_dropped.

A tunnel inside a tunnel. Each layer takes its share: GRE inside IPsec leaves around 1400. Lower the inner MTU by hand, or clamp the MSS on the router where traffic enters the tunnel.

UDP that loses big messages. MSS only helps TCP. DNS answers with DNSSEC, or VPNs over UDP, rely on fragmentation, and one lost fragment loses the whole datagram.

Questions people ask

What is the difference between MTU and MSS? MTU is the largest IP packet a link carries, headers included. MSS is the largest TCP payload in one segment, announced in the SYN: MTU minus 40 bytes for IPv4, 60 for IPv6.

Should I raise my MTU to 9000? Only when every device on the link and every hop of the path agrees. Jumbo frames help inside a data centre or a storage network; on the internet the path is 1500 or less anyway.

How do I find the MTU of a path? Ping with Don't Fragment and shrink the size until it passes: ping -M do -s 1472, then lower. tracepath does the same hop by hop and prints pmtu for each.

See it on real machines