Virtual Networks, Routing, and Queues

Small packets pass through, but slightly larger ones get stuck; veth interfaces show traffic, but the physical port does not. Virtual devices, Layer 2 domains, routing, encapsulation, and queuing need to be explained layer by layer along the same actual path.

1. Bridge: FDB is not a routing table

The bridge learns ports based on source MAC addresses and queries the FDB (Forwarding Database) for entries corresponding to destination MAC addresses. Known unicast traffic is forwarded to the corresponding port; unknown unicast, broadcast, and multicast traffic are handled according to configuration, such as flooding or multicast listening. It does not select the next hop based on the longest IP prefix match.

Assume VMs A and B are connected to two TAP ports on br0, located in the same Layer 2 domain. A first obtains B's link-layer address via ARP/NDP, then sends a frame with B's MAC as the destination. The bridge's learning and forwarding do not require br0 to have an IP address itself; assigning an IP to br0 allows the host to participate in that Layer 3 network, but it is not a prerequisite for enabling bridging functionality.

Layer 2 loops can cause broadcast storms, requiring correct topology and mechanisms such as STP. When VLAN filtering is enabled, port membership, PVID, and tagged/untagged egress rules together determine the forwarding domain; simply creating a VLAN ID does not mean all ports allow that VLAN. See the bridge documentation.

2. VLAN and bonding do not add the same type of capability

VLANs distinguish logical Layer 2 domains over shared physical links. VLAN sub-interfaces usually present independent net_devices to upper layers; hardware offload may store tags in metadata or have hardware insert/strip them, so whether packet captures see tags depends on the location.

Bonding organizes multiple lower-layer links into a single logical interface:

Two 10 Gbit/s links forming an LACP bond do not guarantee that a single TCP connection will reach 20 Gbit/s; to maintain order, the same flow usually still maps to a single member link. Link detection does not equal end-to-end service health detection. See the bonding documentation.

3. Tunnels: Calculate encapsulation first, then configure MTU

IPIP places an IP packet inside an outer IP header; GRE adds a GRE header and can carry corresponding protocols; VXLAN often encapsulates Layer 2 frames into UDP/IP, with the VNI identifying the logical network. A larger VNI compared to a VLAN ID does not imply encryption or access authentication.

Assume the underlay IPv4 MTU is 1500, with standard VXLAN and no extra options:

When the outer header changes to an IPv6 basic header, it typically uses 20 bytes, resulting in approximately 1430. Inner VLANs or additional headers must be counted further.

This is the upper limit calculated based on explicit encapsulation, not a fixed MTU for all VXLANs. The underlay MTU is defined by its IP payload boundaries; the above calculation does not deduct the outer Ethernet header again; mixing layers may result in over-deduction or under-deduction.

Inconsistent MTUs in the path, filtered ICMP Packet Too Big messages, etc., can create a phenomenon where small packets pass normally but large ones get stuck. Tunnel keys/VNIs, remote addresses, FDB/control plane, and return paths also need to be correct. WireGuard and others provide encrypted tunnels, but their security attributes differ from plaintext VXLAN. See the VXLAN documentation.

Change IPv4/IPv6 and VLAN, then watch each header consume the same length budget.

$$D_{\max }={\mathrm{MTU}}_{\mathrm{outer}}-H_{\mathrm{tunnel}}-H_{\mathrm{inner}\mathrm{IP}+\mathrm{TCP}}$$

MTU starts at the outer IP packet. Tunnel overhead includes outer IP/UDP/VXLAN and inner Ethernet; outer Ethernet is outside this IP MTU.

4. Policy routing selects rules first, then checks corresponding tables

The routing policy database checks conditions such as source address, mark, and interface according to rule priority, and performs actions such as table lookups. If a table selected by a lookup rule does not yield a terminating result, it may continue to subsequent rules; it does not unconditionally stop at the first matching condition.

Within the same table, routes usually prioritize the longest prefix, then select based on candidate attributes. 10.2.3.9 matches both 10.0.0.0/8 and 10.2.0.0/16; if other conditions are normal, the latter is more specific. You cannot promise strict O(log n) under any load just because the IPv4 FIB uses a trie.

from ipaddress import ip_address, ip_network

address = ip_address('10.2.3.9')
routes = [ip_network(s) for s in ('0.0.0.0/0', '10.0.0.0/8', '10.2.0.0/16')]
best = max((r for r in routes if address in r), key=lambda r: r.prefixlen)
assert str(best) == '10.2.0.0/16'
print(best)

This example only demonstrates prefix matching within one table and does not simulate policy rules, ECMP, route types, or neighbor resolution. ip route get also needs to include source addresses, marks, or interface information relevant to the problem to approximate actual lookup conditions. See ip-rule.

5. qdisc: Queuing, scheduling, shaping, and AQM

They can be combined, but this does not mean one interface automatically provides all functions.

fq_codel combines flow queues with CoDel; CAKE combines shaping, fairness, and queue management; HTB supports hierarchical bandwidth allocation. The system default qdisc is affected by configuration and devices; it is incorrect to uniformly claim fq_codel is always the default.

Controlling queue position: Local AQM cannot manage all remote buffers

If the real bottleneck is in the modem/ISP buffer, the host's local queue may always be short, and local AQM cannot directly control remote queues. Setting a controllable shaping rate at an appropriate bottleneck before the remote buffer gives the chance to have queuing occur in a managed position, but specific rates require measuring link overhead and variations.

ingress/clsact provide ingress filtering positions, not ordinary receive shaping queues that can queue for long periods; when queuing and shaping are needed, methods such as IFB can be combined. BQL adjusts in-flight bytes at the driver level, which is different from flow-level congestion control and qdisc AQM.

6. Namespaces, veth, TUN/TAP

Network namespaces isolate network states such as device lists, routing, and port bindings. They share the same kernel and do not automatically create egress, DNS services, or bandwidth guarantees. A regular net_device belongs to one namespace at a time; the two ends of a veth pair can be placed in different namespaces.

veth provides paired kernel network endpoints; TUN/TAP allows user-space to read and write network data via file descriptors. TUN provides Layer 3 packets; TAP provides Layer 2 frames. User-space reads from the fd to get data sent by the kernel to that interface, and writes to the fd to inject receiving data into the kernel; the direction should be understood from this perspective.

Whether data carries extra packet information or virtio-net headers depends on flags, not always starting from the first byte as an IP/Ethernet header. VMs commonly use TAP; some user-space VPNs use TUN; the mainline kernel's WireGuard implements its own net_device, so it is incorrect to broadly say all WireGuard implementations rely on /dev/net/tun. See TUN/TAP.

7. Read-only reference for the complete path

ip -d link show
ip rule show
ip route show table all
bridge link show
bridge vlan show
tc -s qdisc show

First confirm devices and namespaces, then check Layer 2 domains, addresses, and routing, followed by confirming MTU, filter rules, and queues. Traffic appearing on veth but not on the physical port is not sufficient to directly determine packet loss: it may be delivered locally, forwarded within the bridge, encapsulated to another egress, or rejected early by policy.

Related reading: Protocol Stack Overview, Netfilter and BPF/XDP.