Ethernet and MAC

When a host accesses a server in another subnet, the Ethernet destination MAC is usually the gateway interface, not the remote server. Understanding this allows you to separate "whether the link is up," "whether neighbors are resolved," and "whether routing is correct." Ethernet provides frame carrying for the local link; bridging and VLANs determine the Layer 2 forwarding scope, while upper-layer routing determines the next hop.

Distinguish Responsibilities of Ethernet, Bridging, and Physical Layer

IEEE 802.3 covers Ethernet MAC and various physical layers; 802.1Q handles bridging/VLAN capabilities, and link aggregation falls under 802.1AX. Do not call all of these frame format extensions of 802.3. Copper or fiber, encoding, and rates vary, yet the upper layers can still use similar MAC services.

Modern common switched full-duplex links do not use CSMA/CD, nor do they have collision processes under normal operation. The background of historical shared media/half-duplex minimum frames and collision detection helps understand evolution, but cannot be used to explain all throughput bottlenecks today; under full duplex, congestion mainly manifests as queuing, dropping, or flow control. Cisco: Full Duplex and Collision Detection

Understand Ethernet II Frame Fields One by One

The following lengths are calculated from the destination address, excluding preamble and SFD. The unmarked frame header is 14 B, and FCS is 4 B; a common 1500 B MTU represents the amount of L3 data that can be carried, so the max-length MAC frame is 1518 B. Do not treat MTU as the total byte count including the Ethernet header.

1501～1535 cannot be casually interpreted as normal EtherType or length; the length form also involves encapsulations like LLC/SNAP and should not be collectively called "raw mode." Common EtherTypes include IPv4 0x0800, ARP 0x0806, IPv6 0x86DD, MAC Control 0x8808, and Slow Protocols 0x8809. PAUSE uses 0x8808, and LACP belongs to 0x8809; do not mix them up. IANA IEEE 802 Parameters Registry

The basic Ethernet preamble is usually recorded as seven 0x55, and SFD is 0xD5; represented with least significant bit first, the final boundary pattern differs from the alternating preamble. Do not say that modern SerDes no longer needs frame boundary mechanisms, nor equate the on-wire representation after PHY encoding simply with the original octet string.

FCS uses a specified CRC algorithm, covering from Dest MAC to the end of data/padding; if a tag is present, it includes the tag, excluding preamble and SFD. It detects transmission errors, does not provide source identity authentication, and does not retransmit on its own. Padding is used to meet the minimum length and does not belong to IP Total Length; drivers should avoid leaking uninitialized memory and should not treat random memory padding as a normal implementation strategy.

Why Small Packets Hit Packet Rate Bottlenecks First

Based on the common budget of 8 B for preamble/SFD per frame and an average 12 byte-times IFG, a 64 B minimum MAC frame occupies 84 byte-times; at 1 Gb/s, this is approximately 1.488 Mframe/s. A 1518 B max-length frame occupies 1538 byte-times, and the ideal efficiency for 1500 B L3 load is 1500/1538≈97.53%. If we further assume IPv4 and TCP headers are each 20 B with no options, the ideal upper limit for 1460 B TCP data is about 949.3 Mb/s, not yet accounting for ACK traffic and host limits.

IFG is the inter-frame idle time. 96 bit times at 1 Gb/s is 96 ns, and at 10 Gb/s is 9.6 ns; some high-speed interfaces have implementation details like word alignment and deficit idle count. Do not rigidly apply a fixed 12 bytes in a single observation without considering PHY/MAC rules. Devices process in parallel via pipelines, caches, DMA, and queues, and do not need to complete all software processing of a frame within one IFG.

Large frames reduce per-byte per-packet overhead, but interrupt coalescing, NAPI, and segmentation/coalescing offloading change the workload observed by the host. Therefore, do not say that increasing MTU from 1500 to 9000 necessarily reduces interrupts by six times.

On the same wire, why do small packets exhaust processing capacity more easily?

Converting the time required for a single transmission into byte-times allows comparing frame rate and effective throughput with the same denominator. When dragged to minimum packets, TCP data can even be zero, yet the link still pays the full cost of the minimum frame. The ribbon chart allocates widths according to true proportions; the table below retains values for small fields.

$$F=\max (64,14+L_{IP}+4),PPS=\frac{R}{8(F+20)},G=R\frac{L_{IP}-40}{F+20}$$

MAC Address is a Link Identifier, Not a Trusted Identity

Common MACs are 48 bits. Using the lowest bit number of the first byte as bit 0, bit 0=1 indicates a group address, and bit 1=1 indicates local management. For example, 02:00:00:00:00:01 is locally managed unicast; ff:ff:ff:ff:ff:ff is broadcast. Traditional OUI allocation commonly uses the first three bytes, but IEEE allocation blocks have different lengths; do not infer a unique vendor from the first three bytes of any address.

Local management, virtualization, randomization, and configuration errors can all produce duplicates or changes, and addresses can be spoofed. A switch learning a MAC only provides forwarding table information and does not mean the device has been authenticated. RFC 9542: IEEE 802 Identifiers and Allocation

Broadcast usually floods only to ports within the same VLAN that meet forwarding state and policy, not unconditionally copying to all switch ports; reserved multicast addresses have special handling. Ordinary switches generally learn the ingress based on source MAC and look up the forwarding table based on destination MAC and VLAN; unknown unicast handling, multicast listening, and port isolation also change the actual egress.

The router receives the link frame destined for itself, removes the old Layer 2 encapsulation, selects the egress based on IP routing, and then generates a new frame using the next-hop MAC of that egress. In the example without NAT, the final destination IP remains unchanged, but TTL/Hop Limit is decremented, and relevant header fields are updated per protocol. See ARP and NDP for MAC next-hop resolution.

Across a router, which addresses change?

First fix the final communicating parties, then observe the addresses actually needed on each link. Setting the initial TTL to 1 and selecting the next link segment reveals that the result should be "no packet," rather than a frame with TTL=0 that continues moving forward. This boundary tests both encapsulation replacement and forwarding lifecycle.

VLAN Tag: Read TCI in Network Byte Order First

The 802.1Q tag is located after the source MAC, including a 2 B TPID and 2 B TCI, followed by the inner EtherType. When decoding TCI as a 16-bit unsigned integer: PCP=(TCI>>13)&7, DEI=(TCI>>12)&1, VID=TCI&0x0fff. The positions from left to right in the diagram are opposite to the numbering direction starting from the least significant bit of the integer; implementations should extract fields using the masks above.

When PCP=5, DEI=1, VID=100, TCI=0xB064, and the two bytes on the wire are B0 64. VID=0 does not mean a normal VLAN 0 exists; the ingress still needs to determine the affiliated VLAN according to port classification rules. Untagged traffic can also be classified by the ingress port's PVID.

Adding a tag usually brings a frame carrying 1500 B payload to 1522 B; L3 MTU does not need to be reduced because of this. Whether short frames retain original padding and the length after inserting/removing tags must be analyzed according to specific sending and bridging behaviors; do not force a minimum payload value on all tagged frames.

Q-in-Q commonly uses outer S-tag 0x88A8 and inner C-tag 0x8100, in the order of DA, SA, outer tag, inner tag, upper-layer type, and data. It extends carrying and isolation identifiers but does not equal encryption. Linux QoS to PCP mapping and VLAN offloading configurations also need to be checked separately; do not assume that reading PCP automatically yields the expected queue policy.

MTU and Packet Capture: Distinguish Three Views

Jumbo MTU like 9000 is a common configuration, not consistently guaranteed by all devices; the end-to-end path must meet corresponding frame or packet capabilities, and tunnels also need to reserve encapsulation overhead. Layer 2 devices dropping frames due to being too large typically do not generate ICMP for routers; Layer 3 MTU, fragmentation, and PMTUD are another set of mechanisms, with different behaviors for IPv4 and IPv6.

Software packet capture may not see preamble or FCS; VLAN tags may be stripped by hardware and delivered as metadata; send checksums may not yet be completed by the NIC, and TSO/GSO or GRO may make software objects larger than on-wire frames. When seeing "oversized MTU packets" or "bad checksums," first check the capture location and offloading, rather than immediately asserting line violations. Linux Segmentation Offloads

Linux's rx-fcs requests retaining received FCS, while rx-all relates to error frames that are usually dropped upon reception; support depends on the NIC/driver. Do not mix them into one switch, and tcpdump will not magically obtain content of frames already dropped by hardware. Linux netdev features

The fields and bitmasks in this article are also cross-checked with Linux 7.2's include/uapi/linux/if_ether.h and include/linux/if_vlan.h; kernel length constants clearly distinguish whether FCS is included.

PAUSE, PFC, and LACP Solve Different Problems

PAUSE is per-link backpressure, requesting the peer to pause corresponding normal sends, with time units in 512 bit times. It can suppress instantaneous pressure on the receiving side, but may also cause unrelated traffic to wait together. PFC distinguishes pauses by priority, but may still produce head-of-line blocking, congestion propagation, or even deadlocks under specific dependencies; do not describe it as the default superior solution for all data center traffic, nor generalize a specific RoCE deployment strategy into a mandatory condition for all networks.

LACP is responsible for negotiating aggregation members and states, not for automatically splitting one connection into multiple paths. Commonly, per-flow hashing is used to avoid per-packet out-of-order delivery, so an aggregation composed of two 10 Gb/s links can increase total capacity for multiple flows, but a single flow is usually still limited by one member link; hash collisions may also cause multiple flows to land on the same member. Linux bonding documentation

Fast/Slow LACP common send periods are about 1 s/30 s and should not be directly taken as the final failover time. Link detection, timeouts, aggregation selection, switch-side configuration, and traffic redistribution jointly determine recovery. Modes like active-backup and balance-rr also have different peer requirements and order impacts; do not configure it as "just plug in two cables."

Local Resolution and Wire-Speed Recalculation

from math import isclose
import struct

pcp, dei, vid = 5, 1, 100
tci = (pcp<<13)|(dei<<12)|vid
assert tci == 0xB064
wire = struct.pack('!HHH', 0x8100, tci, 0x0800)
tpid, received, etype = struct.unpack('!HHH', wire)
assert tpid == 0x8100 and etype == 0x0800
assert ((received>>13)&7, (received>>12)&1, received&0xfff) == (5,1,100)
first = 0x02
assert (first&1) == 0 and (first&2) != 0
assert 1_488_095 < 1e9/((64+8+12)*8) < 1_488_096
assert isclose(1500/1538, 0.9752925877763329)
assert 949.2 < 1e3*1460/1538 < 949.3
assert isclose(96/1e9*1e9, 96)
assert isclose(1*512/1e9*1e6, 0.512)
print('TCI byte order/bit fields, MAC flags, and frame time budget checks passed')

This article only verifies format and mathematical budgets, without modifying interface MAC, MTU, VLAN, or aggregation configurations, and without performing actual packet captures.

Continue Reading

• ARP and NDP: How IP next-hops map to link addresses.
• Switching, Bridging, and VXLAN: Learning tables, broadcast domains, loops, and overlay networks.
• ICMP and PMTUD: Path MTU and error feedback.