---
title: TCP/IP Protocol Stack
url: https://doc.liz6.com/en/linux-kernel/06-network-subsystem/02-TCP-IP
locale: en
area: linux-kernel
tags:
- linux-kernel
- network-subsystem
date: 2026-06-30
modified: 2026-07-16
description: 'Coverage: IPv4/IPv6 receive/transmit paths → TCP state machine → congestion control (CUBIC/BBR) → flow control (sliding window) → socket buffer (sk_wmem/sk_rmem) → TCP fast path Kernel versions: 2.6 ~ 6.x'
---

# TCP/IP Protocol Stack

> Coverage: IPv4/IPv6 receive/transmit paths → TCP state machine → congestion control (CUBIC/BBR) → flow control (sliding window) → socket buffer (sk_wmem/sk_rmem) → TCP fast path
> Kernel versions: 2.6 ~ 6.x

## Overview

The Linux TCP/IP stack is one of the most highly optimized subsystems in the kernel. In the fast path (established sockets, in-order packets), processing each packet takes only a few dozen instructions. This article focuses on the TCP receive fast path and congestion control.

## IPv4 Receive and Transmit

### ip_rcv → ip_local_deliver (Receiving)

```c
// net/ipv4/ip_input.c
ip_rcv()
  ├─ Validation: version==4, checksum OK, valid length
  ├─ netfilter NF_INET_PRE_ROUTING (iptables raw/mangle)
  └─ ip_rcv_finish()
      ├─ Route lookup: fib_lookup() → dst_entry
      │   → RTN_LOCAL: for local delivery → ip_local_deliver()
      │   → RTN_UNICAST: forwarding → ip_forward()
      └─ ip_local_deliver()
          ├─ netfilter NF_INET_LOCAL_IN (iptables filter)
          └─ ip_local_deliver_finish()
              └─ ipprot->handler → tcp_v4_rcv() / udp_rcv()
```

### ip_queue_xmit (Transmitting)

```c
// net/ipv4/ip_output.c
__ip_queue_xmit(sk)
  ├─ Route lookup: ip_route_output_ports() → dst_entry + source IP
  ├─ Construct IP header: protocol, TTL, DF flag, ... 
  ├─ netfilter NF_INET_LOCAL_OUT
  └─ ip_local_out()
      ├─ netfilter NF_INET_POST_ROUTING
      └─ dev_queue_xmit() → QoS qdisc → NIC driver ndo_start_xmit()
```

## TCP Fast Path: tcp_rcv_established

```c
// net/ipv4/tcp_input.c
// This is the hottest path for TCP packet reception (established socket, in-order packets)

tcp_rcv_established(sk, skb)
  // fast path (90%+ of packets take this route):
  ├─ Check header prediction:
  │   ├─ PSH bit set? → skip
  │   ├─ In-order? (seq == rcv_nxt) ✓
  │   ├─ Window non-zero? ✓
  │   └─ URG/RST/SYN? → skip
  │
  ├─ Copy data to receive queue: tcp_queue_rcv()
  │   └─ skb_copy_datagram_msg() → directly to user buffer? (if MSG_DONTWAIT)
  │
  ├─ Update rcv_nxt, window advertisement
  │
  └─ If data is in queue → sk_data_ready() → wake up epoll/read

  // slow path:
  └─ OOO (out-of-order) / SACK / window full / FIN / RST → tcp_validate_incoming()
```

## TCP State Machine

<svg viewBox="0 0 720 420" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,'Source Han Sans CN','Microsoft YaHei',sans-serif" role="img" aria-label="TCP State Machine: From Three-Way Handshake to Active/Passive Close">
  <defs>
    <marker id="tcpFsmArrow" markerWidth="10" markerHeight="8" refX="8" refY="3" orient="auto"><path d="M0,0 L8,3 L0,6 Z" fill="#475569"/></marker>
  </defs>
  <rect width="720" height="420" fill="#ffffff"/>
  <text x="360" y="28" text-anchor="middle" font-size="17" font-weight="700" fill="#1f2933">TCP State Machine: From Three-Way Handshake to Active/Passive Close</text>

  <!-- row1: Connection Establishment Path -->
  <rect x="20" y="54" width="90" height="36" rx="8" fill="#e2e8f0"/>
  <text x="65" y="77" text-anchor="middle" font-size="12" font-weight="700" fill="#334155">CLOSED</text>
  <rect x="140" y="54" width="100" height="36" rx="8" fill="#eef2ff" stroke="#c7d2fe"/>
  <text x="190" y="77" text-anchor="middle" font-size="12" font-weight="700" fill="#3730a3">LISTEN</text>
  <rect x="270" y="54" width="110" height="36" rx="8" fill="#eef2ff" stroke="#c7d2fe"/>
  <text x="325" y="77" text-anchor="middle" font-size="12" font-weight="700" fill="#3730a3">SYN_RECV</text>
  <rect x="410" y="54" width="150" height="36" rx="8" fill="#4f46e5"/>
  <text x="485" y="77" text-anchor="middle" font-size="12" font-weight="700" fill="#ffffff">ESTABLISHED</text>

  <line x1="110" y1="72" x2="137" y2="72" stroke="#475569" stroke-width="1.6" marker-end="url(#tcpFsmArrow)"/>
  <text x="125" y="46" text-anchor="middle" font-size="10.5" fill="#64748b">bind + listen</text>
  <line x1="240" y1="72" x2="267" y2="72" stroke="#475569" stroke-width="1.6" marker-end="url(#tcpFsmArrow)"/>
  <text x="255" y="46" text-anchor="middle" font-size="10.5" fill="#64748b">accept</text>
  <line x1="380" y1="72" x2="407" y2="72" stroke="#475569" stroke-width="1.6" marker-end="url(#tcpFsmArrow)"/>
  <text x="395" y="46" text-anchor="middle" font-size="10.5" fill="#64748b">SYN</text>

  <!-- ESTABLISHED splits into two paths -->
  <line x1="485" y1="90" x2="310" y2="147" stroke="#475569" stroke-width="1.6" marker-end="url(#tcpFsmArrow)"/>
  <text x="395" y="118" text-anchor="middle" font-size="10.5" fill="#64748b">active close (FIN)</text>
  <line x1="485" y1="90" x2="617" y2="147" stroke="#475569" stroke-width="1.6" marker-end="url(#tcpFsmArrow)"/>

  <!-- row2: Close Branches -->
  <rect x="250" y="150" width="110" height="36" rx="8" fill="#ffedd5"/>
  <text x="305" y="173" text-anchor="middle" font-size="12" font-weight="700" fill="#9a3412">FIN_WAIT1</text>
  <rect x="390" y="150" width="110" height="36" rx="8" fill="#ffedd5"/>
  <text x="445" y="173" text-anchor="middle" font-size="12" font-weight="700" fill="#9a3412">FIN_WAIT2</text>
  <rect x="540" y="150" width="160" height="44" rx="8" fill="#ffedd5"/>
  <text x="620" y="169" text-anchor="middle" font-size="12" font-weight="700" fill="#9a3412">CLOSE_WAIT</text>
  <text x="620" y="184" text-anchor="middle" font-size="9.5" fill="#c2410c">(Remote closed, local not yet closed)</text>

  <line x1="360" y1="168" x2="387" y2="168" stroke="#475569" stroke-width="1.6" marker-end="url(#tcpFsmArrow)"/>
  <text x="373" y="160" text-anchor="middle" font-size="10.5" fill="#64748b">ACK</text>

  <!-- row3: Finalization -->
  <rect x="250" y="230" width="110" height="36" rx="8" fill="#ffedd5"/>
  <text x="305" y="253" text-anchor="middle" font-size="12" font-weight="700" fill="#9a3412">CLOSING</text>
  <rect x="390" y="230" width="140" height="36" rx="8" fill="#f0fdfa" stroke="#99f6e4"/>
  <text x="460" y="253" text-anchor="middle" font-size="12" font-weight="700" fill="#115e59">TIME_WAIT (2MSL)</text>
  <rect x="580" y="230" width="110" height="36" rx="8" fill="#ffedd5"/>
  <text x="635" y="253" text-anchor="middle" font-size="12" font-weight="700" fill="#9a3412">LAST_ACK</text>

  <line x1="305" y1="186" x2="305" y2="227" stroke="#475569" stroke-width="1.6" marker-end="url(#tcpFsmArrow)"/>
  <line x1="445" y1="186" x2="459" y2="227" stroke="#475569" stroke-width="1.6" marker-end="url(#tcpFsmArrow)"/>
  <line x1="360" y1="248" x2="387" y2="248" stroke="#475569" stroke-width="1.6" marker-end="url(#tcpFsmArrow)"/>
  <line x1="620" y1="194" x2="635" y2="227" stroke="#475569" stroke-width="1.6" marker-end="url(#tcpFsmArrow)"/>

  <!-- row4: Passive Close Final State -->
  <rect x="580" y="310" width="110" height="36" rx="8" fill="#e2e8f0"/>
  <text x="635" y="333" text-anchor="middle" font-size="12" font-weight="700" fill="#334155">CLOSED</text>
  <line x1="635" y1="266" x2="635" y2="307" stroke="#475569" stroke-width="1.6" marker-end="url(#tcpFsmArrow)"/>

  <rect x="20" y="356" width="680" height="50" rx="8" fill="#eef2ff" stroke="#c7d2fe"/>
  <text x="36" y="376" font-size="12.5" fill="#3730a3">After ESTABLISHED, there are two paths: active close goes through FIN_WAIT1→FIN_WAIT2→TIME_WAIT(2MSL), or simultaneous close goes through CLOSING;</text>
  <text x="36" y="396" font-size="12.5" fill="#3730a3">Passive close (remote sends FIN first) goes through CLOSE_WAIT→LAST_ACK→CLOSED.</text>
</svg>

## Congestion Control

```c
// net/ipv4/tcp_cong.c
// Pluggable congestion control modules:
struct tcp_congestion_ops {
    void (*cong_avoid)(struct sock *sk, u32 ack, u32 acked);
    void (*ssthresh)(struct sock *sk);    // Reset slow start threshold on packet loss
    u32  (*undo_cwnd)(struct sock *sk);   // False retransmission detection → rollback cwnd
};

// CUBIC (default):
//   cwnd = C * (t - K)^3 + W_max
//   K = cube_root(W_max * beta / C)
//   Advantages: Fast recovery, fairness, good performance on high-speed links

// BBR (Bottleneck Bandwidth and RTT, Google):
//   Not loss-based, but based on BDP (Bandwidth-Delay Product)
//   Measurements: max BW (last 10 rounds) and min RTT (last 10 seconds)
//   Goal: Keep inflight ≈ BDP
//   Advantages: Significantly outperforms CUBIC on bufferbloat-prone links
```

## TCP Socket Buffer Tuning

```bash
# Send buffer (default 16KB, auto-grows to max)
cat /proc/sys/net/ipv4/tcp_wmem  # min default max
# Receive buffer
cat /proc/sys/net/ipv4/tcp_rmem

# Auto-tuning (enabled by default):
#   sk_wmem and sk_rmem auto-grow based on RTT and bandwidth probing
#   → High BDP links do not require manually setting large buffers
```

## Debugging

```bash
# TCP connection states
ss -tianp   # Includes cwnd, rtt, rcv_mss, and all statistics

# Congestion control
sysctl net.ipv4.tcp_congestion_control
cat /proc/sys/net/ipv4/tcp_available_congestion_control

# Packet loss/retransmission statistics
nstat -a | grep -E 'TcpRetrans|TcpExt'
cat /proc/net/snmp | grep Tcp

# Trace TCP state changes
bpftrace -e 'kprobe:tcp_set_state { printf("%s: %d -> %d\n", comm, arg1, arg2); }'
```

## References

- **Source Code**: `net/ipv4/tcp_input.c`, `net/ipv4/tcp_output.c`, `net/ipv4/tcp_cong.c`, `net/ipv4/tcp_cubic.c`, `net/ipv4/tcp_bbr.c`
- **RFCs**: RFC 793 (TCP), RFC 5681 (Congestion Control), RFC 8312 (CUBIC)
- **LWN**: "TCP fast path", "BBR congestion control"

*Keywords: tcp_rcv_established, fast path, CUBIC, BBR, congestion control, socket buffer, TCP state machine*