Network Protocol Stack Overview

Coverage: sk_buff → NAPI → softirq (NET_RX/NET_TX) → protocol stack layers → socket API → net_device → multi-queue Kernel version: 2.6 ~ 6.x

Overview

The Linux network stack is the most widely used TCP/IP implementation in the world. From NIC DMA packet reception to the user-space read() return, packets go through 10+ layers of processing. This article traces this complete path, focusing on core data structures and design decisions.

sk_buff: Packet Descriptor

// include/linux/skbuff.h
struct sk_buff {
    struct sk_buff      *next, *prev;  // Doubly linked list (scheduling queue)
    struct sock         *sk;           // Associated socket

    ktime_t             tstamp;        // Timestamp (SO_TIMESTAMPING)
    struct net_device   *dev;          // Input/output device

    // Data pointers (linear data + non-linear frags)
    unsigned char       *head, *data;
    unsigned int        len, data_len;
    sk_buff_data_t      tail, end;

    // Protocol header pointers (set during layer-by-layer parsing)
    __be16              protocol;      // L3 (ETH_P_IP, ETH_P_IPV6)
    unsigned char       *transport_header;  // L4 (TCP/UDP)
    unsigned char       *network_header;    // L3 (IP header)
    unsigned char       *mac_header;        // L2 (Ethernet)

    // Segmentation/Reassembly
    unsigned int        gso_size, gso_segs;  // GSO: Large packets → hardware segmentation
    unsigned short      nr_frags;            // Number of non-contiguous pages
};

The core design of sk_buff is the four-pointer layout of head/data/tail/end: data resides in the buffer allocated from head to end, while data to tail represents valid data. The regions head to data and tail to end are headroom and tailroom, respectively. As the protocol stack adds each layer's header, the data pointer moves backward, reusing the existing headroom—no copying is required.

NAPI: Hybrid Interrupt + Polling

// net/core/dev.c
// Problem: High-speed NICs (10Gbps+) receive millions of packets per second → one interrupt per packet → interrupt storm
// NAPI: First packet triggers an interrupt notification → interrupts disabled → polling for packet reception in softirq

// NIC Driver:
//   1. Packet received → disable RX interrupt → napi_schedule(&napi)
//   2. NET_RX_SOFTIRQ triggered → napi_poll()
//      → Batch receive packets (one budget, typically 300 packets)
//      → After processing a batch → if more packets remain → continue polling
//      → If no more packets → enable RX interrupt → return to step 1

Complete Packet Reception Path

flowchart TD
    DMA["NIC DMA<br/>→ Ring Buffer (RX ring)"]
    DMA -->|"Hardware Interrupt"| IRQ["Driver Interrupt Handler<br/>napi_schedule()"]
    IRQ -->|"NET_RX_SOFTIRQ"| POLL["napi_poll()<br/>Batch receive, up to 300 packets"]

    POLL -->|"Per Packet"| NETIF["__netif_receive_skb()"]

    NETIF --> XDP["XDP hook (bpf)<br/>Earliest processing point"]

    NETIF --> RXH{"RX Handler<br/>List Check"}
    RXH -->|"tcpdump"| AF_PACKET["AF_PACKET<br/>Copy to packet socket"]
    RXH -->|"bridge/bonding/VLAN"| REDIR["Possible Redirection"]

    NETIF --> DELIVER["deliver_skb()"]

    DELIVER --> IP_RCV["ip_rcv()<br/>L3 (IPv4)"]

    IP_RCV --> IP_LOCAL["ip_local_deliver()"]

    IP_LOCAL --> NF{"netfilter<br/>PREROUTING"}
    NF -->|"FORWARD"| FWD["Forward to other interfaces"]
    NF -->|"INPUT"| IP_FINISH["ip_local_deliver_finish()"]

    IP_FINISH --> TCP_RCV["tcp_v4_rcv()<br/>L4 (TCP)"]

    TCP_RCV --> TCP_DO["tcp_v4_do_rcv()"]

    TCP_DO --> STATE{"Connection State?"}
    STATE -->|"ESTABLISHED"| FAST["tcp_rcv_established()<br/>Fast path ⚡"]
    STATE -->|"LISTEN"| SLOW["tcp_rcv_state_process()"]
    SLOW --> QUEUE["Data enqueued to receive queue"]

    FAST --> QUEUE
    QUEUE --> WAKE["sk_data_ready()<br/>Wake up blocked read() ✅"]

    classDef hw fill:#fff3e0,stroke:#ef6c00
    classDef irq fill:#ffebee,stroke:#c62828
    classDef l2 fill:#e3f2fd,stroke:#1565c0
    classDef l3 fill:#f3e5f5,stroke:#7b1fa2
    classDef l4 fill:#e8f5e9,stroke:#2e7d32
    classDef decision fill:#fff8e1,stroke:#f9a825
    class DMA,IRQ hw
    class POLL,NETIF,XDP,RXH,AF_PACKET,REDIR l2
    class IP_RCV,IP_LOCAL,IP_FINISH,DELIVER,NF,FWD l3
    class TCP_RCV,TCP_DO,FAST,SLOW,QUEUE,WAKE l4
    class STATE decision

Multi-Queue and RSS

RSS (Receive Side Scaling):
  NIC uses hash(src_ip, dst_ip, src_port, dst_port) % N
  → N receive queues → N CPUs

  NAPI's napi_struct is per-queue:
    → Each queue has its own NAPI → Parallel packet reception
    → Packets from the same flow always go to the same queue → TCP ordering preserved

XPS (Transmit Packet Steering):
  Multi-queue on the transmit side: similar concept but for TX

net_device: Network Device Abstraction

// include/linux/netdevice.h
struct net_device {
    char                name[IFNAMSIZ];    // eth0, wlan0
    const struct net_device_ops *netdev_ops;
    // → ndo_open, ndo_stop, ndo_start_xmit, ndo_set_rx_mode, ...

    struct net_device_stats stats;         // Statistics filled by driver
    struct list_head    qdisc_list;        // Queuing disciplines (tc)

    unsigned int        num_rx_queues;
    unsigned int        num_tx_queues;
    struct netdev_rx_queue  *_rx;          // RX queue array (per-queue NAPI)
};

Debugging

# Packet receive/transmit statistics
cat /proc/net/dev
ip -s link

# Softirq statistics (NET_RX/NET_TX)
cat /proc/softirqs | grep NET

# NAPI statistics (per-queue polling)
ethtool -S eth0 | grep -E 'rx_packets|rx_bytes|tx_'

# Drop reasons (drop counter)
dropwatch -l kas

References

Source Code: net/core/dev.c, include/linux/skbuff.h, include/linux/netdevice.h
Kernel Documentation: Documentation/networking/
LWN: "The Linux networking stack", "NAPI design"

Keywords: sk_buff, NAPI, NET_RX_SOFTIRQ, multi-queue, RSS, net_device, XDP