Network Protocol Stack Overview

Application send success, NIC transmission completion, peer ACK, and business processing completion are distinct moments. Along the transmit/receive path, you must track in-flight packets, skbs, and application byte streams separately; they do not necessarily correspond one-to-one.

1. skbs are descriptors, not fixed contiguous packets

struct sk_buff stores metadata and references actual data buffers. The linear region is described by the logical positions of head, data, tail, and end; some members store offsets in the implementation and are not all directly dereferenceable C pointers.

skb->len is the total data length, data_len is the non-linear part length, and skb_headlen() returns the linear valid length. The header members mainly represent relative positions; access should use appropriate interfaces like ip_hdr() and tcp_hdr(). GSO information and frags are located in structures like skb_shared_info; do not arbitrarily flatten them into a fabricated skb definition.

During packet reception parsing, skb_pull() advances the data pointer; during encapsulation, skb_push() utilizes headroom to prepend headers. However, these operations are only valid if space, linear range, and shared state conditions are met; calling them does not automatically expand capacity or establish copy-on-write semantics.

Reference counting maintains lifetime but does not grant permission for arbitrary concurrent modifications. See skb documentation.

2. How received data reaches the application

DMA writes frames into buffers managed by the driver; completion descriptors inform the driver which data is available for processing. Drivers must also adhere to DMA cache, ordering, and buffer recycling rules; discovering that a packet has been received does not mean the page can be immediately returned to the NIC for overwriting.

Common native XDP runs before constructing standard skbs. Packet socket capture entry points, bridge RX handlers, and VLAN processing belong to different mechanisms and cannot all be drawn as a single RX handler list; the position of generic XDP is also not the same as native XDP.

IP PREROUTING occurs before routing decisions. Local delivery continues through INPUT, while forwarding goes through FORWARD and egress processing. Placing PREROUTING after ip_local_deliver() would directly mislead DNAT and routing analysis.

TCP reception also requires checking connections, sequences, windows, checksums, and states; data is queued into the corresponding receive structure before notifying waiters. sk_data_ready() wake-ups do not mean the application has been scheduled to run, nor that the application has copied or processed all data. MSG_DONTWAIT only changes waiting behavior; it does not give softirqs a special path to automatically copy directly to the current user buffer.

3. NAPI: Processing time via budget allocation

NAPI handles reception and transmission completion via poll methods, often running in softirqs, and also supports threaded NAPI and busy polling. In traditional interrupt-driven modes, interrupts trigger scheduling while keeping relevant device interrupts masked; after appropriate work is done, polling completes and notifications are restored.

Budget: View individual polls and the entire NET_RX cycle separately

The budget in the poll callback limits the RX workload for this round and should not be confused with the netdev_budget for the entire NET_RX cycle as "fixed 300 packets per NIC per time." TX completion cleanup does not simply consume the same RX budget. Calls with a budget of 0 have specific restrictions and cannot perform RX/page-pool/XDP operations or call completion interfaces.

Exhausting the budget usually implies that polling must continue; if processing finishes exactly when the work count equals the budget, it must still be handled according to API conventions to avoid false completion reports or missed subsequent processing. The successful return of napi_complete_done() participates in the decision to restore device interrupts.

NAPI instances, IRQs, and RX/TX queue pairs often have a near one-to-one organizational mapping, but the protocol does not mandate this mapping. One instance can serve multiple queues, and transmission/reception can be separated. See NAPI documentation.

Change the budget and add packets. Pay attention to an empty queue after exactly exhausting the budget: emptiness alone does not prove scheduling ownership was released.

Trade-offs between throughput and latency

Increasing batch sizes, hardware interrupt coalescing, or polling can amortize notification costs; however, it may also cause the first packet to wait longer or allow a busy queue to occupy the CPU for longer. Budget exhaustion should be judged in conjunction with softirq time, queue backlog, packet drops, and application latency; you cannot simply assert faster performance just because there are fewer interrupts.

4. Two meanings of transmission completion

Application send success usually indicates that data has been accepted by the socket or queued into the transmit path. TCP is also subject to window, congestion control, and pacing constraints, then constructs an skb, passes through routing, Netfilter, qdisc, and driver transmission.

NIC completion does not cross the latter two boundaries. When troubleshooting latency, you must clarify which type of completion you are waiting for.

Network drivers provide device operations via interfaces like net_device_ops; statistics are often exported via interfaces like rtnl_link_stats64. Virtual devices also implement net_device, so a single skb may pass through multiple net_devices; do not interpret "reached net_device" as having reached the physical NIC.

5. Why GRO, GSO, and TSO change packet capture perception

GRO merges compatible packets on the receive side, reducing the number of protocol-processing operations; GSO describes how to segment larger skbs later, which software can complete; TSO is hardware offload for TCP segmentation. GSO does not mean "only hardware segmentation," and GRO does not unconditionally merge any same-connection data into a single packet.

Capture position: Objects before segmentation are not equal to in-flight frames

Therefore, the "large TCP packet" captured locally may just be the skb before segmentation; in-flight traffic is still sent according to the path MTU. When the transmit-side checksum is yet to be completed by hardware, packet captures showing incomplete checksums are not necessarily line errors. Verification must occur at the correct observation point; do not adjust MTU or disable all offloads based solely on local captures. See Segmentation Offloads.

6. RSS, RPS, RFS, XPS: What to choose for each

RSS often maps via hash and indirection tables and cannot be simply defined as a four-tuple hash modulo the number of queues. Configuration, encapsulation, fragmentation, and NIC capabilities affect which fields participate in the hash; modifying indirection or mapping may also cause traffic migration.

Stable mapping for the same flow aids in-order delivery and cache hits but does not replace TCP's own sequence handling. Increasing the number of queues also increases interrupt, memory, and CPU migration costs. See Network Scaling Mechanisms.

7. Read-only observation of queues and counters

from pathlib import Path

for dev in sorted(Path('/sys/class/net').glob('*')):
    names = sorted(p.name for p in (dev / 'queues').glob('*'))
    counters = {}
    for key in ('rx_packets', 'tx_packets', 'rx_dropped', 'tx_dropped'):
        try:
            counters[key] = int((dev / 'statistics' / key).read_text())
        except OSError:
            pass
    print(dev.name, names, counters)

These are cumulative counts from the device perspective; virtual and physical devices may count the same traffic, and GRO/GSO as well as driver definitions affect observation. /proc/softirqs statistics count processing instances, not packet counts; private fields in ethtool -S are not unified NAPI statistics interfaces.

Take the difference between two time points and correlate it with the actual load window. For finer-grained root causes, check driver counters, socket statistics, or controlled tracing, rather than interpreting all rx_dropped as the NIC RX ring being full.