Link aggregation

More capacity and no single point of failure, without the loop two plain parallel links would make. Each packet is put on one member by a hash of its headers, so a flow always takes the same cable and never arrives out of order: layer2 hashes MAC addresses, layer3+4 IP addresses and ports. One flow never gets more than one cable's speed; many flows add up. cat /proc/net/bonding/bond0 shows a Linux bond's members and state.

How link aggregation works, step by step

Two plain cables between the same two switches make a loop, and spanning tree blocks one of them. Link aggregation (IEEE 802.1AX, first published as 802.3ad) joins them into one logical link instead: a bond on Linux, a port channel or EtherChannel on Cisco, a LAG in the standard. It has one MAC address and, on a router or server, one IP address. The cables are its members.

1. Both ends are configured with the same members. With LACP (the Link Aggregation Control Protocol) they also check each other: every member sends a LACPDU to 01:80:c2:00:00:02, a slow-protocols address that switches never forward. The default rate is one every 30 seconds; the fast rate is one per second.
2. Each LACPDU describes both ends: the actor (the sender's system ID, key and port) and the partner (what it has heard from the other end). A member joins the bond only when both ends agree it belongs.
3. The member's state flags then read Synchronization, Collecting, Distributing: in sync, accepting traffic, sending traffic.
4. Each outgoing frame takes one member, chosen by a hash of its headers. All frames of one flow give the same hash and take the same cable, so they never arrive out of order. Each end hashes what it sends, independently.
5. When a member fails, its flows are rehashed onto the rest. A dead carrier is noticed in milliseconds. A cable that stays up but carries nothing is noticed by LACP when three LACPDUs are missed: 3 seconds at the fast rate, 90 at the slow one.

The hash decides how well the load spreads. Hashing MAC addresses (layer2) sends everything between two routers over one cable. Hashing IP addresses and ports (layer3+4) gives each connection its own chance. Either way, one flow never gets more than one cable's speed: a bond adds lanes, not speed.

What it looks like on a Linux machine

In the link aggregation lab a router and a file server are joined by two 100 Mbit/s cables in an LACP bond. The members carry no address of their own:

$ ip -br link show master bond0
bond0p1          UP             9e:3b:52:c1:07:4a <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP>
bond0p2          UP             9e:3b:52:c1:07:4a <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP>

Both have the bond's MAC. The kernel's own report shows the mode, the hash and each member:

$ cat /proc/net/bonding/bond0
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
...
802.3ad info
LACP active: on
LACP rate: fast
...
Slave Interface: bond0p1
MII Status: up
Speed: 100 Mbps
...
Aggregator ID: 1

With the default layer2 policy, four parallel iperf3 connections still get 100 Mbit/s: every frame has the same two MAC addresses, so all four ride one cable. After ip link set dev bond0 type bond xmit_hash_policy layer3+4 at both ends, they spread and get 200 Mbit/s together, while a single connection stays at 100. Pull a cable during a transfer and the throughput halves, but no connection breaks.

What goes wrong, and how you see it

One cable busy, the other idle. The hash policy groups all the traffic into one flow from the bond's point of view. Watch per-member counters (ip -s link or ifstat) and change the policy.

A member that never joins. Mismatched speed or duplex, different keys, or LACP at one end only. The member's partner state stays incomplete in /proc/net/bonding/bond0, and on a switch it shows suspended or individual.

A static bond and a silent failure. Without LACP (balance-xor on Linux, mode on on Cisco), a cable that breaks behind a patch panel keeps its carrier at one end. That end keeps sending half its flows into it, and those connections hang. LACP's missed LACPDUs catch it.

Members to two different switches. A normal bond needs both ends on one device. Spreading members over two switches needs MLAG, vPC or a stack that presents them as one partner.

Questions people ask

Does link aggregation double the speed? For many flows together, up to the sum of the members. For a single flow, no: it stays on one cable.

What is the difference between LACP and a static LAG? Both spread traffic the same way. LACP adds the handshake, so a miscabled or silently failed member is kept out instead of black-holing traffic.

Do I still need spanning tree? STP sees the bond as one link, so it blocks none of its members. Keep STP on for the rest of the network.

See it on real machines