Network Performance and Tuning

From application buffers to TCP congestion control to NIC offload—every layer has tunable parameters. BDP determines the buffer size needed to saturate bandwidth; BBR outperforms CUBIC on bufferbloat-prone links by using a model instead of packet loss detection; fq_codel/cake actively manage queues to prevent excessive buffer filling.

Overview

Most "TCP tuning secrets" circulating online are just sysctl settings piled up without considering the context. Effective tuning follows only one path: first, understand which queues data passes through from the application to the wire, identify the bottleneck layer, and then adjust the knobs for that specific layer. Every stop along a packet's transmission path can become a bottleneck, and each has its own tuning approach:

flowchart LR
    APP["📦 Application buffer<br><small>write block size</small>"] --> SK["Socket buffer<br><small>tcp_wmem / tcp_rmem</small>"]
    SK --> CC["TCP Congestion Control<br><small>cubic / bbr</small>"]
    CC --> QD["qdisc Queue<br><small>fq_codel / cake</small>"]
    QD --> DRV["Driver Ring Buffer<br><small>ethtool -G</small>"]
    DRV --> NIC["NIC Offload<br><small>TSO / GSO / GRO</small>"]
    NIC --> NET(("🌐 Network<br><small>You can't control queues here<br>→ bufferbloat</small>"))
    classDef knob fill:#4493f81f,stroke:#4493f8,stroke-width:2px
    classDef wild fill:#f8514926,stroke:#f85149,stroke-width:2.5px
    class APP,SK,CC,QD,DRV,NIC knob
    class NET wild

Three typical problems correspond to three different layers: underutilized bandwidth is usually due to socket buffers being smaller than the BDP or an unsuitable congestion algorithm; spikes in latency during load are queue issues (bufferbloat); low small-packet throughput/CPU saturation are interrupt and offload issues. These are detailed below.

Underutilized Bandwidth: BDP and Buffer Sizing

For TCP to saturate a link, the unacknowledged data "in flight" must fill the entire pipe. The pipe capacity is BDP (Bandwidth × RTT), and the in-flight limit is constrained by min(cwnd, receive window, send buffer)—if any of these three is smaller than the BDP, bandwidth will not be fully utilized.

# Triplet: min / default / max — auto-tuning only scales between min and max
sysctl net.ipv4.tcp_rmem="4096 131072 67108864"   # Receive side, 64MB max
sysctl net.ipv4.tcp_wmem="4096 16384 67108864"    # Send side
sysctl net.core.rmem_max=67108864                  # Upper limit for manual SO_RCVBUF setting

Verification method: Use ss -ti to check cwnd, rtt, and bytes_acked for a single connection. If cwnd stays pinned to the buffer limit rather than fluctuating with the link, the buffer is the bottleneck.

Note: Buffers are not "the bigger, the better." Large buffers on low-BDP links just waste memory, and on devices that are themselves bottlenecks, they exacerbate queuing delay. Set them according to the maximum BDP you need to support, leaving a 2x margin.

Latency Spikes Under Load: Bufferbloat and Queue Management

Classic symptom: Ping is 10ms when idle, but jumps to 500ms as soon as a download starts. The cause is not bandwidth, but queues—some device in the path (often a home router or optical modem) is configured with a huge FIFO buffer. Loss-driven algorithms like CUBIC will keep sending packets until it fills up, causing all traffic to queue in this deep buffer.

On the ends you can control, replace FIFO with AQM (Active Queue Management):

tc qdisc replace dev eth0 root fq_codel              # Default on most distros
tc qdisc replace dev eth0 root cake bandwidth 950mbit # First choice for home egress

• fq_codel = Fair Queuing + CoDel. CoDel's approach is to monitor the residence time of packets: if it consistently exceeds 5ms, it starts dropping packets (from the head of the queue, sending a fast signal) to force senders to slow down—shorter queues mean lower latency. The fair queuing component ensures one elephant flow doesn't starve an adjacent SSH session.
• cake is an engineering improvement over fq_codel: it includes a shaper (set bandwidth slightly below the real upstream capacity to reclaim queue control from the optical modem), per-host isolation (one device running BT won't affect the whole household), and DiffServ awareness.

The best results are achieved when combined with bbr on the sending side—BBR itself does not fill queues (see the congestion control section in TCP Deep Dive):

sysctl net.ipv4.tcp_congestion_control=bbr

Common TCP Tuning Knobs

Principle: Test with load after every change. Knobs interact with each other; changing everything at once is equivalent to changing nothing.

Small Packets and CPU: NIC Offload

At 10Gbps, packet rates can reach millions per second. Processing interrupts per packet will exhaust CPU before bandwidth is saturated. The solution is batching—moving segmentation/merging tasks from the kernel to the NIC:

ethtool -k eth0 | grep -v fixed     # View current offload status
ethtool -K eth0 tso off             # Temporarily disable when troubleshooting
ethtool -G eth0 rx 4096 tx 4096     # Driver ring buffer; increase when dropping packets during burst traffic

A classic tcpdump trap: Seeing 64KB "jumbo frames" in captures is not abnormal—it means TSO/GRO is active, and the capture point is before segmentation/after merging; actual transmission on the wire uses MTU-sized packets. Conversely, if you suspect offload is causing issues (especially checksum errors in virtualized environments), first try ethtool -K ... off to rule it out.

Order for Locating Bottlenecks

1. Baseline: iperf3 -c <server> -R to measure bidirectional bandwidth, getting the raw bandwidth—first confirm whether the bottleneck is indeed in the network.
2. Measure latency under load: Run iperf3 and ping simultaneously. If latency increases tenfold → bufferbloat, adjust queues.
3. Inspect single connections: Check cwnd/rtt/retrans in ss -ti. If cwnd cannot increase, check buffers and congestion algorithm; if retransmissions are high, check link quality.
4. Inspect the system: nstat -a | grep -i retrans, ethtool -S eth0 | grep -i drop (NIC layer drops), softnet_stat backpressure.
5. Change one variable, then return to step 1.

For systematic packet capture analysis, see tcpdump and Wireshark. For troubleshooting methodologies, see Network Troubleshooting Methodology.

References

• BBR: github.com/google/bbr · RFC 9438
• fq_codel/CoDel: RFC 8290, RFC 8289 · bufferbloat.net
• cake: man tc-cake · Engineering follow-up to RFC 8290
• Brendan Gregg: "Systems Performance" Chapter 10 (Network)

Keywords: BDP, BBR, CUBIC, bufferbloat, fq_codel, cake, AQM, TCP tuning, tcp_rmem, TSO, GSO, GRO, ethtool, iperf3