TCP/IP Protocol Stack

Coverage: IPv4/IPv6 receive/transmit paths → TCP state machine → congestion control (CUBIC/BBR) → flow control (sliding window) → socket buffer (sk_wmem/sk_rmem) → TCP fast path Kernel versions: 2.6 ~ 6.x

Overview

The Linux TCP/IP stack is one of the most highly optimized subsystems in the kernel. In the fast path (established sockets, in-order packets), processing each packet takes only a few dozen instructions. This article focuses on the TCP receive fast path and congestion control.

IPv4 Receive and Transmit

ip_rcv → ip_local_deliver (Receiving)

// net/ipv4/ip_input.c
ip_rcv()
  ├─ Validation: version==4, checksum OK, valid length
  ├─ netfilter NF_INET_PRE_ROUTING (iptables raw/mangle)
  └─ ip_rcv_finish()
      ├─ Route lookup: fib_lookup() → dst_entry
      │   → RTN_LOCAL: for local delivery → ip_local_deliver()
      │   → RTN_UNICAST: forwarding → ip_forward()
      └─ ip_local_deliver()
          ├─ netfilter NF_INET_LOCAL_IN (iptables filter)
          └─ ip_local_deliver_finish()
              └─ ipprot->handler → tcp_v4_rcv() / udp_rcv()

ip_queue_xmit (Transmitting)

// net/ipv4/ip_output.c
__ip_queue_xmit(sk)
  ├─ Route lookup: ip_route_output_ports() → dst_entry + source IP
  ├─ Construct IP header: protocol, TTL, DF flag, ... 
  ├─ netfilter NF_INET_LOCAL_OUT
  └─ ip_local_out()
      ├─ netfilter NF_INET_POST_ROUTING
      └─ dev_queue_xmit() → QoS qdisc → NIC driver ndo_start_xmit()

TCP Fast Path: tcp_rcv_established

// net/ipv4/tcp_input.c
// This is the hottest path for TCP packet reception (established socket, in-order packets)

tcp_rcv_established(sk, skb)
  // fast path (90%+ of packets take this route):
  ├─ Check header prediction:
  │   ├─ PSH bit set? → skip
  │   ├─ In-order? (seq == rcv_nxt) ✓
  │   ├─ Window non-zero? ✓
  │   └─ URG/RST/SYN? → skip
  │
  ├─ Copy data to receive queue: tcp_queue_rcv()
  │   └─ skb_copy_datagram_msg() → directly to user buffer? (if MSG_DONTWAIT)
  │
  ├─ Update rcv_nxt, window advertisement
  │
  └─ If data is in queue → sk_data_ready() → wake up epoll/read

  // slow path:
  └─ OOO (out-of-order) / SACK / window full / FIN / RST → tcp_validate_incoming()

TCP State Machine

Congestion Control

// net/ipv4/tcp_cong.c
// Pluggable congestion control modules:
struct tcp_congestion_ops {
    void (*cong_avoid)(struct sock *sk, u32 ack, u32 acked);
    void (*ssthresh)(struct sock *sk);    // Reset slow start threshold on packet loss
    u32  (*undo_cwnd)(struct sock *sk);   // False retransmission detection → rollback cwnd
};

// CUBIC (default):
//   cwnd = C * (t - K)^3 + W_max
//   K = cube_root(W_max * beta / C)
//   Advantages: Fast recovery, fairness, good performance on high-speed links

// BBR (Bottleneck Bandwidth and RTT, Google):
//   Not loss-based, but based on BDP (Bandwidth-Delay Product)
//   Measurements: max BW (last 10 rounds) and min RTT (last 10 seconds)
//   Goal: Keep inflight ≈ BDP
//   Advantages: Significantly outperforms CUBIC on bufferbloat-prone links

TCP Socket Buffer Tuning

# Send buffer (default 16KB, auto-grows to max)
cat /proc/sys/net/ipv4/tcp_wmem  # min default max
# Receive buffer
cat /proc/sys/net/ipv4/tcp_rmem

# Auto-tuning (enabled by default):
#   sk_wmem and sk_rmem auto-grow based on RTT and bandwidth probing
#   → High BDP links do not require manually setting large buffers

Debugging

# TCP connection states
ss -tianp   # Includes cwnd, rtt, rcv_mss, and all statistics

# Congestion control
sysctl net.ipv4.tcp_congestion_control
cat /proc/sys/net/ipv4/tcp_available_congestion_control

# Packet loss/retransmission statistics
nstat -a | grep -E 'TcpRetrans|TcpExt'
cat /proc/net/snmp | grep Tcp

# Trace TCP state changes
bpftrace -e 'kprobe:tcp_set_state { printf("%s: %d -> %d\n", comm, arg1, arg2); }'

References

• Source Code: net/ipv4/tcp_input.c, net/ipv4/tcp_output.c, net/ipv4/tcp_cong.c, net/ipv4/tcp_cubic.c, net/ipv4/tcp_bbr.c
• RFCs: RFC 793 (TCP), RFC 5681 (Congestion Control), RFC 8312 (CUBIC)
• LWN: "TCP fast path", "BBR congestion control"

Keywords: tcp_rcv_established, fast path, CUBIC, BBR, congestion control, socket buffer, TCP state machine