TCP/IP Protocol Stack

The boundaries of send do not become the boundaries of recv verbatim, and ACKs do not imply that the peer has committed business data. TCP provides an ordered, reliable byte stream. When understanding the Linux implementation, first separate sequences, windows, recovery, and shutdown directions.

For packet headers, options, and protocol-level window calculations, start with TCP in the networking series. This article follows Linux connection queues, receive/send paths, and socket behavior; the shared sequence and window models connect the protocol to its implementation.

1. IP Routing Determines Local Delivery or Forwarding

After verifying the basic header, IPv4 undergoes processing such as PREROUTING before making a routing decision. Packets destined for the local machine enter LOCAL_IN/INPUT and are handed to TCP, UDP, etc.; the forwarding path goes through FORWARD and egress processing. Local sending requires initial routing information; address/mark changes in OUTPUT may trigger re-routing, after which the packet enters POSTROUTING.

IPv6 has its own input and routing implementation: the basic header lacks an IPv4 header checksum, and extension header processing and fragmentation rules differ. IPv6 routers do not fragment on behalf of the source along the path; therefore, mechanisms like Path MTU Discovery must be handled correctly. Do not simply treat IPv4 function chains as a universal protocol stack.

2. The Three-Way Handshake Establishes Connection State

A normal active open goes through SYN_SENT, then advances to ESTABLISHED upon receiving a valid response. The server's listening socket continues to listen; half-open connections are typically tracked using a lighter-weight request socket, and only after the handshake is complete do they enter the accept queue. SYN cookies, TCP Fast Open, etc., further alter specific resources and timing.

The SYN queue and the accept queue are at different stages. listen(backlog) does not set a single total count for all half-open connections. If accept returns too slowly, it may block the queue of completed handshakes; this cannot be solved solely by increasing SYN-related parameters.

The handshake exchanges initial sequence numbers and options, and the SYN itself consumes one sequence number. A successful TCP establishment only indicates that the transport connection is available, not that the application has authenticated or that business services are ready. See RFC 9293 for protocol specifications.

3. Sequences, ACKs, and the Receive Queue

TCP sequence numbers count bytes. SACK can report discrete received ranges separately, allowing the sender to choose retransmissions more accurately; cumulative ACKs do not skip gaps that have not yet been filled.

The peer acknowledging bytes entering the receive protocol state does not mean the peer application has read them, nor does it mean business writes to the database have been committed. send returning does not equal receiving an ACK. If the application requires message boundaries or completion acknowledgments, it must define length, delimiters, request IDs, and response protocols.

Linux can place in-order data into the receive queue and store out-of-order data in corresponding structures; ordered byte streams are delivered semantically when the application reads. PSH does not constitute a reliable message boundary; recv(4096) may return fewer than 4096 bytes, or it may merge data from multiple sends in one go.

Linux retains ordered and out-of-order data in corresponding structures and delivers an ordered byte stream. PSH is not a reliable message delimiter. recv(4096) can return fewer bytes or combine data from several sends. Click arbitrary segments to introduce reordering and duplicates; application consumption is counted separately.

4. Flow Control and Congestion Control Are Two Different Rulers

The receive window (rwnd) indicates how much data the receiver can still accept; the congestion window (cwnd) indicates the sender's limit on the amount of in-flight data the network can bear. Their units may differ in implementation and display; for example, cwnd is often counted in MSS segments, requiring conversion before comparison.

Roughly speaking, the amount of new data that can continue to be sent is limited by min(cwnd_bytes, rwnd_bytes) - bytes_in_flight, while also considering pacing, whether the application has data, retransmissions, and implementation details. The window is not a direct unit of sending rate.

Shrinking a limit does not erase already outstanding bytes. Change each limit and then try sending.

$$new\_data\le \max (0,\min (\mathrm{cwnd},\mathrm{rwnd})-\mathrm{flight})$$

All quantities are bytes. This compares current sending allowance, omitting congestion-algorithm evolution, SACK pipe estimates, and the real window-update protocol.

Calculating In-Flight Data: Bandwidth May Be Sufficient, But the Window May Still Limit Throughput

Other limitations are ignored here. The calculation shows that insufficient window size limits throughput; it does not imply that increasing buffers solves all performance issues.

Increasing buffers can eliminate certain limitations but cannot fix packet loss, incorrect MTU, congestion, or slow applications. Receive auto-tuning has upper limits, memory pressure, and user setting constraints; setting parameters like SO_RCVBUF also affects auto-tuning behavior. One cannot promise that "high BDP never requires checking buffers." See TCP sysctl.

5. Loss Detection Is Not Just Timeout

Information such as duplicate ACKs, SACK, and time-based inference via RACK can detect loss before the RTO expires. RTO remains an important fallback mechanism for recovery, based on smoothed RTT and variance estimation, with backoff after timeout.

RFC 6298 provides the RTO calculation framework; specific Linux minimums, paths, and configurable items should be checked against the target version. Do not treat RFC recommended values as fixed values for all Linux timers. RTT samples after retransmission are ambiguous; estimation cannot directly attribute any ACK to the most recent send.

Timers Classified by Purpose: Window, Idle, and Loss

Zero-window probes handle cases where the receiver's window is closed; keepalive checks the reachability of long-idle connections. Neither is a rename of the standard loss retransmission timer. TCP retry timeouts also do not guarantee that the peer application hasn't already processed previous data; business retries require their own deduplication semantics.

6. CUBIC and BBR: Compare Control Objectives, Don't Accept Marketing Claims

CUBIC's window growth is designed around the window value before the last congestion event; the cubic function alters growth behavior as it approaches or moves away from the old peak. It also includes details such as TCP-friendly behavior. You cannot replace the entire algorithm with a single cubic formula. RFC 9438 is the specification for CUBIC, not for BBR.

BBR constructs a path model using estimated delivery rate and propagation RTT, controlling sending via pacing and in-flight limits. Different BBR versions handle packet loss, ECN, fairness, and probing differently; you cannot uniformly say it "completely ignores packet loss" or that it "is always better than CUBIC on any congested link." The actual kernel module name does not automatically prove the adoption of a specific paper/patch version.

cat /proc/sys/net/ipv4/tcp_congestion_control
cat /proc/sys/net/ipv4/tcp_available_congestion_control
ss -tin

The system default algorithm affects new connections; existing sockets can have their own settings. When comparing, fix the path, RTT, number of flows, receive window, and bottleneck queue, then observe throughput, latency, and co-existing flow performance.

7. The Fast Path Does Not Bypass Protocol Correctness

tcp_rcv_established() contains optimized processing for common established connection packets. When conditions like header prediction are met, general state processing overhead is reduced; out-of-order packets, abnormal flags, window changes, options, etc., cause processing to take other branches.

You cannot deduce from the function name that "90% of packets execute only dozens of instructions." Checksum offload, GRO, lock contention, socket backlog, memory allocation, and wake-ups all affect cost. When a socket is occupied by user context, packets may be queued in the backlog and subsequently processed by the appropriate path.

MSG_DONTWAIT only controls whether the current read/write operation waits; it cannot be treated as a condition for the kernel to directly write received packets into the user buffer. The source code should be read in conjunction with net/ipv4/tcp_input.c, tcp.c, and tcp_output.c.

8. Closing One Direction Does Not Mean Closing the Entire Connection

TIME_WAIT and FIN_WAIT2 are different states; do not assume tcp_fin_timeout is a universal switch for TIME_WAIT duration. Long-term accumulation of CLOSE_WAIT often requires checking application close behavior; a large number of TIME_WAIT states cannot be judged as a leak solely by quantity.

Runnable Half-Close Example

The following uses only local loopback and temporary ports. The client closes the sending direction after writing; the server can still return the count of received bytes after seeing EOF.

import socket
import threading

errors = []
with socket.socket() as listener:
    listener.bind(('127.0.0.1', 0))
    listener.listen(1)
    listener.settimeout(3)

    def serve():
        try:
            conn, _ = listener.accept()
            with conn:
                conn.settimeout(3)
                size = 0
                while data := conn.recv(1024):
                    size += len(data)
                conn.sendall(str(size).encode('ascii'))
        except Exception as exc:
            errors.append(exc)

    worker = threading.Thread(target=serve)
    worker.start()
    try:
        with socket.create_connection(listener.getsockname(), timeout=3) as client:
            client.sendall(b'hello')
            client.shutdown(socket.SHUT_WR)
            parts = []
            while data := client.recv(1024):
                parts.append(data)
            assert b''.join(parts) == b'5'
    finally:
        worker.join(timeout=4)
    assert not worker.is_alive()
    if errors:
        raise errors[0]
print('half-close: reply received after SHUT_WR')

This protocol deliberately uses EOF as the request boundary; long-lived multiplexed connections generally require setting length or frame formats separately. It illustrates TCP half-close semantics, but does not simulate network disconnection, retransmission, or application idempotency.