Ethernet and MAC

Ethernet frames are the starting point for "encapsulating everything"—from MAC addresses to VLANs to link aggregation, Layer 2 is the foundational carrier for all upper-layer protocols. Understanding frame structure and switching principles means understanding "how packets get from one wire to another."

Overview

Ethernet is the most widely deployed LAN technology globally. From Bob Metcalfe's 2.94Mbps prototype at Xerox PARC in 1973 to today's 800Gbps optical ports, the core frame format has remained virtually unchanged. It operates at OSI Layer 2 (Data Link Layer), responsible for delivering frames from a source MAC address to a destination MAC address within the same broadcast domain. The 802.3 standard family extends this with features such as VLAN isolation (802.1Q), flow control (802.3x PAUSE), and link aggregation (802.3ad LACP). The Linux kernel provides protocol processing and hardware driver abstractions through net/ethernet/ and drivers/net/.

802.3 Frame Format

The Ethernet frame structure has undergone virtually no essential changes since the DIX specification in 1980. The only notable evolution is the insertion of the 802.1Q VLAN tag and the negotiation layer for 802.3ad LACP.

Octet 0-6:   Preamble (7 × 0x55)
              Used by the PHY layer for clock synchronization and bit-locking.
              Extremely important at 10Mbps (Manchester encoding),
              though modern SerDes does not require it, it has not been removed.

Octet 7:     SFD (Start Frame Delimiter = 0xD5)
              Marks the start of the frame. The least significant bit of the last
              preamble byte is different — 0x55 → 0xD5 (bit 0: 1→1, bit 2: 1→0) —
              hardware detects this transition to lock onto byte boundaries.

Octet 8-13:  Destination MAC (6 bytes)
              Bit 0 of byte 0: 0 = unicast, 1 = multicast.
              All 1s (ff:ff:ff:ff:ff:ff) = broadcast — switch floods to all ports.

Octet 14-19: Source MAC (6 bytes)

Octet 20-21: EtherType or Length (2 bytes)
              ≤ 1500 (0x05DC): Interpreted as Length (802.3 raw mode, rarely used)
              ≥ 1536 (0x0600): Interpreted as EtherType
              Key values:
                0x0800  IPv4
                0x0806  ARP
                0x8100  VLAN (802.1Q)
                0x86DD  IPv6
                0x88A8  Q-in-Q / Provider Bridge (802.1ad)
                0x8870  Jumbo frame (non-standard, used by some drivers)
                0x8809  Slow Protocols (LACP, 802.3x PAUSE)

Octet 22+:   Payload
              Standard: 46-1500 bytes (excluding VLAN tag).
              If less than 46 bytes → padded to 46 (padding). Padding content is undefined —
              Linux uses zero padding, while other drivers may use random data.
              Padding is not included in the IP header's Total Length — the receiver
              distinguishes it via the EtherType/Length field or the L3 header length.

CRC:         FCS (Frame Check Sequence, 4 bytes, CRC-32)
              Polynomial: x³² + x²⁶ + x²³ + x²² + x¹⁶ + x¹² + x¹¹ + x¹⁰ + x⁸ + x⁷ +
                       x⁵ + x⁴ + x² + x + 1
              Calculation range: Destination MAC to end of Payload (excluding Preamble/SFD/FCS itself)
              Hardware (MAC controller) calculates and verifies → erroneous frames are
              discarded by hardware, so the kernel never sees them.
              Some drivers can be configured for "receive error frames" mode (ethtool -K rx-fcs on),
              used for debugging (tcpdump can see FCS error counts).

Inter-Frame Gap (IFG):
  Inter-frame gap: 12 byte times (96 bit times) of idle.
  1000BASE-T: 96 ns
  10GBASE-R:  9.6 ns
  100GBASE-R: 0.96 ns
  At high speeds, the IFG is extremely tight — hardware must complete processing
  of the previous frame and start preamble detection for the next frame within one IFG.
  This is why high-speed NICs require buffer descriptor rings rather than per-frame interrupts.

MAC Address Structure

48 bits (6 bytes), IEEE assigned, globally unique (theoretically):

Byte 0, bit 0 (I/G bit):
  0: Unicast (individual address)
  1: Multicast (group address)
  → This is the first step for the MAC layer to determine "is this frame for me" —
     checked by NIC hardware.

Byte 0, bit 1 (U/L bit):
  0: Universally Administered (OUI burned in by manufacturer)
  1: Locally Administered (set by software, e.g., virtual NICs, containers, MAC randomization)

OUI (Organizationally Unique Identifier): bytes 0-2 = IEEE assigned
NIC-specific: bytes 3-5 = manufacturer-assigned

Linux operations:
  ip link set eth0 address aa:bb:cc:dd:ee:ff    ← Change MAC (requires interface to be down)
  ip link show eth0 | grep link/ether

802.1Q VLAN

VLAN tag (4 bytes) is inserted after the Source MAC and before the EtherType:

Octet 0-1: TPID (Tag Protocol Identifier)
             0x8100 = Customer VLAN (C-VLAN)
             0x88A8 = Service Provider VLAN (S-VLAN, 802.1ad)
             0x9100 = Legacy (replaced by 0x88A8)

Octet 2-3: TCI (Tag Control Information):
  Bits 0-2:    PCP (Priority Code Point, 802.1p)
               0 = Best Effort, 1 = Background, 2 = Spare, 3 = Excellent Effort,
               4 = Controlled Load, 5 = Video, 6 = Voice, 7 = Network Control
               → Maps to sk_buff->priority → queue selection in tc qdisc
  Bit 3:       DEI (Drop Eligible Indicator)
               1 = Can be dropped preferentially during congestion (not the same as IP DSCP ECN)
  Bits 4-15:   VID (VLAN Identifier)
               Valid values: 1-4094 (0 = no VLAN, 4095 = reserved)
               Each Ethernet interface can belong to multiple VLANs (trunk port)

Q-in-Q (802.1ad, Provider Bridge):
  Double VLAN tag:
    [DA][SA][0x88A8][S-VLAN TCI][0x8100][C-VLAN TCI][EtherType][Payload]
  Outer (S-VLAN) = Operator network isolation
  Inner (C-VLAN) = Customer's own VLAN

  Linux:
    ip link add link eth0 name eth0.100 type vlan id 100
    ip link add link eth0.100 name eth0.100.200 type vlan id 200 proto 802.1ad

Jumbo Frames

The standard MTU of 1500 bytes is a compromise from the 10Mbps Ethernet era — a small MTU meant a short slot time, and thus a short collision detection window. After switching to switched Ethernet, this physical limitation is no longer needed, but the standard MTU is assumed by all upper-layer protocols as the default.

Jumbo MTU: Typically 9000 bytes (non-standard, but widely supported)

Benefits:
  - Reduced interrupts: 1 × 9000B frame vs 6 × 1500B frames — interrupts reduced by 6x
  - Reduced CPU processing: NIC per-packet overhead (descriptor, DMA sync) reduced by 6x
  - Reduced header overhead: Saves 5 extra frames' worth of (14B Eth + 20B IP + 20B TCP) × 5 = 270B
  - NFS/iSCSI: Most IO is 4KB-64KB, jumbo frames can fit more data in a single frame

Costs:
  - All devices on the path (switches, routers, NICs) must support jumbo frames
  - Devices that do not support jumbo frames: silently drop packets (no ICMP Frag Needed unless IP DF is set)
    → "MTU black hole" — one of the hardest network issues to troubleshoot

Configuration:
  ip link set eth0 mtu 9000
  /etc/dhcpcd.conf or NetworkManager: persist MTU settings
  Note: The bridge's MTU must be ≥ the minimum MTU of all its slave interfaces

PAUSE (802.3x) and PFC (802.1Qbb)

802.3x PAUSE frame:
  EtherType: 0x8808
  MAC Control Opcode: 0x0001 (PAUSE)
  Pause Time: 2 bytes (in units of 512 bit times)
  Sent to multicast address: 01:80:c2:00:00:01

  Receiver: Sends PAUSE frame (pause_time=N)
  Sender: Stops transmitting for N × 512 bit times
  Issue: Congestion in one flow → stops traffic on the entire link
        → Not suitable for scenarios with mixed traffic flows (data centers use PFC)

PFC (Priority-based Flow Control, 802.1Qbb):
  Independent control per PCP (8 priorities):
  Only the congested priority is paused → other priorities continue transmitting
  → Hardware foundation for RoCE (RDMA over Converged Ethernet)

LACP (802.3ad / 802.1AX)

LACP binds multiple physical ports into a single logical link:

LACPDU (Link Aggregation Control Protocol Data Unit):
  EtherType: 0x8809 (Slow Protocols)
  Subtype: 0x01 (LACP)
  Sent every 30 seconds (slow) or every second (fast)

LACPDU Content:
  Actor: system_priority + system_id (MAC) + key + port_priority + port_number + state
  Partner: (same as above, remote information)
  Collector: max_delay

State bits (critical):
  LACP_Activity (bit 0): Active (sends LACPDUs) vs Passive (responds only)
  LACP_Timeout (bit 1): Long (30s) vs Short (1s)
  Aggregation (bit 2): This port can be aggregated
  Synchronization (bit 3): This port is synchronized (ready to forward)
  Collecting (bit 4): Ingress traffic can be forwarded
  Distributing (bit 5): Egress traffic can be forwarded
  Defaulted (bit 6): Using default parameters (partner did not respond)

Linux bonding modes:
  mode=0 (balance-rr): Round-robin — does not depend on LACP, only load balances
  mode=1 (active-backup): Active/standby — does not depend on LACP
  mode=2 (balance-xor): Based on MAC hash — requires switch cooperation
  mode=4 (802.3ad): LACP dynamic negotiation — switch must also have LACP enabled

References

802.3-2022: IEEE Standard (P802.3 Working Group)
Kernel: net/ethernet/eth.c (protocol processing), drivers/net/bonding/ (LACP), net/8021q/ (VLAN)
LWN: "Jumbo frames and MTU in Linux"

Keywords: Ethernet, 802.3, MAC address, 802.1Q, VLAN, Q-in-Q, jumbo frame, PAUSE, LACP, 802.3ad