TCP

The application calls send twice, but the receiver might read all the data at once or in multiple reads. TCP maintains an ordered byte stream and does not preserve the message boundaries written by the application. Understanding it can be done by tracing a single data transfer: sequence numbers identify bytes, ACKs indicate reception progress, the window constrains the sendable range, loss detection triggers recovery, and congestion control determines how much load to impose on the network.

Byte Stream: Reliable reception does not equal business completion

TCP recovers data through acknowledgments, retransmissions, and ordered delivery as long as the connection can be maintained; it cannot guarantee eventual success in the event of a permanent network outage. A successful send usually indicates that the data has entered the local send path; a TCP ACK indicates that the peer TCP has received the corresponding bytes, but it does not prove that the peer application has read, written to disk, or committed the transaction. Business confirmation should still be expressed by the application protocol.

For example, with two writes of abc and def, the receiver might read ab and cdef. Framing such as length prefixes, fixed lengths, or delimiters defines message boundaries; PSH is not a reliable message separation flag. TCP does not natively provide encryption or server identity authentication; mechanisms like TLS take on another layer of responsibility. RFC 9293: TCP Basic Specification

How header fields connect to behavior

TCP length must be derived from the IP layer length and the actual header chain; do not use Data Offset as the length of the entire segment. 32-bit sequence numbers wrap around; comparisons should follow the protocol's modular sequence number rules and window constraints, as standard integer magnitude comparisons cannot cover all states.

Options: Some are only negotiated during connection establishment

Window scaling is directional: the shift advertised by C is used to interpret the receive window sent by C later, limiting S's transmission to C. MSS also constrains data sent by the peer to the local side, and actual transmission is further limited by PMTU, options, congestion, and receive window. RFC 7323: Window Scale and Timestamps

Handshake: Sequence numbers and connection state

The client's SYN uses sequence number 1000, the server acknowledges it with ACK=1001, and proposes its own ISN=5000; the client then acknowledges 5001. The third step allows the server to know that its sequence number has been acknowledged by the peer, which helps exclude old duplicate connection attempts. Handshake acknowledgments reflect the protocol state at that time and do not guarantee that the network will always be reachable afterwards.

SYN and FIN each occupy one sequence number, allowing control events to be uniformly included in ordered acknowledgment and retransmission; this cannot be reversed to mean "messages that do not occupy sequence numbers cannot be retransmitted in any protocol." Pure ACKs do not occupy sequence numbers, nor is there a separate acknowledgment for each ACK.

A normal client goes through SYN-SENT, the server goes through SYN-RECEIVED, and then enters ESTABLISHED; simultaneous open, data-carrying extensions, and abnormal resets have separate rules. SYN cookies are an implementation mechanism to resist certain half-open connection resource pressures; they have negotiation capabilities and usage conditions and cannot replace normal queue planning or all protections. RFC 9293: Connection Establishment

Cumulative ACK, SACK, and Retransmission Intervals

Assume [1001,2001) and [3001,4001) are received, with [2001,3001) lost in between. The cumulative ACK can only be 2001; SACK additionally reports [3001,4001). After filling the gap, the consecutive range extends to 4001, and the ACK can jump there.

SACK expresses reception facts; the sender still needs to maintain a scoreboard, judge loss, and schedule retransmissions. "Enabling SACK means you will always only retransmit the one piece of data that is truly lost" is too absolute; out-of-order delivery and misjudgments affect recovery. The receiver also has boundaries for withdrawing previously SACKed data, so the sender cannot discard all retransmission copies just because of SACK before the cumulative acknowledgment. RFC 2018: SACK

Loss detection and congestion control are related but distinct problems: the former determines which data might be lost, and the latter determines how much load is allowed next. Classic fast retransmission uses signals such as duplicate ACKs; modern RACK infers loss based on send time and subsequent acknowledgments; TLP uses tail probes to improve recovery for certain tail losses, without waiting for three duplicate ACKs. RFC 8985: RACK-TLP

Why RTO cannot directly equal RTT

RTO needs to leave room for delay variation. The basic estimate in RFC 6298 is SRTT + max(G, 4×RTTVAR), combined with minimums and backoff rules. When the first sample R=100 ms, SRTT=100 ms, RTTVAR=50 ms, the original calculated value is 300 ms; according to the 1 s lower bound suggested by this RFC, it becomes 1 s, and the actual kernel's lower bound implementation should be verified separately.

When the next sample is 120 ms, update RTTVAR using the old SRTT first: 0.75×50+0.25×20=42.5 ms; then update SRTT: 0.875×100+0.125×120=102.5 ms. The original RTO=272.5 ms. The ACK after retransmission may not be able to determine which send it is acknowledging; the Karn rule and timestamps are used to handle the corresponding sampling ambiguity; timeouts also trigger backoff. RFC 6298: Retransmission Timers

Where did ACK reach, and what did the app read?

Send the third range, then the first: cumulative acknowledgment advances only over contiguous data. Filling the second can advance ACK by two ranges. Read afterward to distinguish acknowledgment from consumption. Adjacent displayed SACK ranges may be merged.

Flow Control and Congestion Window: Deduct in-flight data first

In a normal, non-recovery simplified model, the sender is constrained by both the receive window rwnd and the congestion window cwnd. min(rwnd,cwnd) is the in-flight size constraint, not that you can send that much every time; you also need to deduct unacknowledged data, and consider pacing, available application data, and actual implementation.

Assume SND.UNA=1001, SND.NXT=4001, then there are already 3000 B unacknowledged. When rwnd=8000 B, cwnd=5000 B, the simplified new data margin is min(8000,5000)−3000=2000 B. After increasing cwnd to 20000 B, the constraint becomes the receive window, and the new margin is 5000 B.

After a zero window, the sender's persist mechanism probes whether the window has reopened, preventing long stalls caused by lost window updates; it serves a different purpose than idle connection keepalive and packet loss retransmission timers. Nagle limits small segment transmission when there is unacknowledged data, which may increase wait times for small interactions when combined with delayed ACKs; TCP_NODELAY does not restore application message boundaries.

Bandwidth-Delay Product Recalculation

For 1 Gbit/s, RTT=50 ms, the BDP is 10⁹/8×0.05=6,250,000 B. With an unscaled 65535 B window, even ignoring other constraints, the upper bound given by window/RTT is only about 10.486 Mbit/s. With scale=7, the maximum representation is 8,388,480 B, approximately 8 MiB, which can just cover this BDP; this is only a necessary budget, not a throughput guarantee.

A window is not remaining send credit

The sliders control distinct state variables. Put flight size between the windows to see why the larger limit cannot compensate for the smaller one. Shrink a limit below flight size: new sending pauses while outstanding bytes remain.

$$credit=\max \left(0,\min (rwnd,cwnd)-FlightSize\right)$$

Teardown: Two directions end independently

After A sends FIN, it no longer sends new application data, but can still receive data from P. After P acknowledges, it may stay in CLOSE-WAIT until the application finishes its own sending direction; then P sends FIN, and A acknowledges. ACK and FIN can be combined, so teardown does not necessarily require exactly four packets. Simultaneous teardown also changes the state path.

A typical active close side goes through FIN-WAIT-1, FIN-WAIT-2, TIME-WAIT; the passive side goes through CLOSE-WAIT, LAST-ACK. A large number of CLOSE-WAIT states should check why the application has not yet closed the connection, rather than explaining it with TIME_WAIT parameters.

TIME-WAIT reserves time to respond to retransmitted FINs and prevent old connection packets from interfering with new connections; it does not guarantee that the final ACK is definitely delivered. The specification uses 2MSL to describe it; the TCP_TIMEWAIT_LEN in Linux 7.2 is 60×HZ (60 s); distinguish the specification’s model from an implementation timer. tcp_fin_timeout applies to corresponding orphan FIN-WAIT-2 states, not as a TIME-WAIT duration switch.

Connection multiplexing can reduce connection establishment/teardown frequency; SO_REUSEADDR affects bindings that meet the conditions and cannot eliminate all four-tuple conflicts. tcp_tw_reuse also has security conditions and version configuration implications; distinguish between state and resource bottlenecks first, then discuss parameters. RFC 9293: Teardown, Linux TCP sysctl

CUBIC: How the retained fraction shapes growth

The network can reflect congestion with signals such as packet loss and ECN; it is not true that congestion control is only "guessing" and the network never actively feeds back. A core target curve for CUBIC is:

W(t) = C×(t−K)³ + Wmax

When β represents the retention ratio after congestion, K = ∛(Wmax×(1−β)/C). Taking β=0.7, C=0.4, Wmax=100 segments, K≈4.22 s; W(0)=70 segments, W(K)=100 segments. The old formula directly put the retention ratio 0.7 into the numerator of K, causing W(0) to incorrectly become 30 segments. RFC 9438: CUBIC

The curve slows down as it approaches the old Wmax and continues to probe after passing it; time is the independent variable of this function, but actual TCP still relies on ACK, RTT, Reno-friendly regions, and recovery rules, and cannot be summarized as throughput being completely unaffected by RTT. The approximate exponential growth at startup is also affected by ACK behavior and limiting conditions, and it is not guaranteed that every connection doubles exactly every RTT.

CUBIC's congestion response does not mean that every individual packet loss independently multiplies by 0.7; event merging, fast recovery, and timeouts have different handling. Queue size, AQM, ECN, random packet loss, and competing flows all affect final latency and throughput.

Why does growth slow then accelerate around the inflection?

Move through K: the window keeps growing while the slope reaches zero then rises. Changing β moves both the initial window and K, while W(K) stays at old Wmax. This exposes the purpose of the horizontal shift.

$$W_{cubic}(t)=C(t-K{)}^3+W_{max},K=\sqrt[3]{W_{max}(1-\beta )/C}$$

BBR: Bandwidth and delay models also need verification

BBR uses observed delivery rate and RTT to estimate path capacity, controlling transmission in combination with pacing and in-flight constraints. The typical process includes rapidly probing capacity, draining queues formed during startup, periodically probing bandwidth, and updating the minimum delay estimate. Observations are affected by ACK aggregation, application constraints, path changes, and other flows; the model does not directly read the true physical bottleneck.

The tcp_bbr.c in Linux 7.2 includes the classic four modes, a startup pacing gain of approximately 2.885, and steady-state gains such as 1.25/0.75; this cannot be generalized as a unified constant for all BBR versions. Bandwidth estimation, cwnd gain, and pacing gain are also different, and the in-flight data cannot always be precisely pinned to one BDP.

Google's BBR project includes continuously evolving implementations and materials; the compared version should be clearly specified. RFC 9438 defines CUBIC; consult the BBR project for the algorithm and version being compared. Do not replace experimental conditions with statements like "BBR never slows down due to packet loss," "must have no queues," or unspecified deployment percentages. Google BBR Project

Local Deduction: Byte Intervals, Windows, and Curves

import math

received = [(1001,2001),(3001,4001)]
def cumulative(intervals, start=1001):
    ack = start
    for left,right in sorted(intervals):
        if left>ack:
            break
        ack = max(ack,right)
    return ack
assert cumulative(received) == 2001
assert cumulative(received+[(2001,3001)]) == 4001
assert min(8000,5000)-(4001-1001) == 2000
assert 1_000_000_000/8*.05 == 6_250_000
assert 65535<<7 == 8_388_480
assert math.isclose(65535*8/.05/1e6,10.4856)

srtt,variation = .1,.05
sample = .12
variation = .75*variation+.25*abs(srtt-sample)
srtt = .875*srtt+.125*sample
assert math.isclose(srtt+4*variation,.2725)

beta,C,Wmax = .7,.4,100
K = (Wmax*(1-beta)/C)**(1/3)
def target(t):
    return C*(t-K)**3+Wmax
assert math.isclose(target(0),70)
assert math.isclose(target(K),100)
assert target(K+1)>100
print('ACK gap, send margin, BDP, RTO, and CUBIC boundary checks passed')

This is a local mathematical model and does not simulate real retransmission timers, kernel congestion states, or network fairness; congestion control and connection parameters are not modified.

Continue Reading

• UDP and QUIC Transport Layer: Message boundaries, multi-streaming, and recovery mechanisms.
• ICMP and PMTUD: Path packet length constraints.
• Network Performance and Tuning: Locating bottlenecks from observational evidence.