---
title: Network Performance and Tuning
url: https://doc.liz6.com/en/networking/13-network-diagnostics/03-network-performance-and-tuning
locale: en
area: networking
tags:
- networking
- network-diagnostics
date: 2026-06-30
modified: 2026-07-16
description: From application buffers to TCP congestion control to NIC offload—every layer has tunable parameters. BDP determines the buffer size needed to saturate bandwidth; BBR outperforms CUBIC on bufferbloat-prone links by using a model instead of packet loss detection; fq_codel/cake actively manage queues to prevent excessive buffer filling.
---

# Network Performance and Tuning

> From application buffers to TCP congestion control to NIC offload—every layer has tunable parameters. BDP determines the buffer size needed to saturate bandwidth; BBR outperforms CUBIC on bufferbloat-prone links by using a model instead of packet loss detection; fq_codel/cake actively manage queues to prevent excessive buffer filling.

## Overview

Most "TCP tuning secrets" circulating online are just sysctl settings piled up without considering the context. Effective tuning follows only one path: **first, understand which queues data passes through from the application to the wire, identify the bottleneck layer, and then adjust the knobs for that specific layer**. Every stop along a packet's transmission path can become a bottleneck, and each has its own tuning approach:

```mermaid
flowchart LR
    APP["📦 Application buffer<br><small>write block size</small>"] --> SK["Socket buffer<br><small>tcp_wmem / tcp_rmem</small>"]
    SK --> CC["TCP Congestion Control<br><small>cubic / bbr</small>"]
    CC --> QD["qdisc Queue<br><small>fq_codel / cake</small>"]
    QD --> DRV["Driver Ring Buffer<br><small>ethtool -G</small>"]
    DRV --> NIC["NIC Offload<br><small>TSO / GSO / GRO</small>"]
    NIC --> NET(("🌐 Network<br><small>You can't control queues here<br>→ bufferbloat</small>"))
    classDef knob fill:#4493f81f,stroke:#4493f8,stroke-width:2px
    classDef wild fill:#f8514926,stroke:#f85149,stroke-width:2.5px
    class APP,SK,CC,QD,DRV,NIC knob
    class NET wild
```

Three typical problems correspond to three different layers: **underutilized bandwidth** is usually due to socket buffers being smaller than the BDP or an unsuitable congestion algorithm; **spikes in latency during load** are queue issues (bufferbloat); **low small-packet throughput/CPU saturation** are interrupt and offload issues. These are detailed below.

## Underutilized Bandwidth: BDP and Buffer Sizing

For TCP to saturate a link, the unacknowledged data "in flight" must fill the entire pipe. The pipe capacity is **BDP (Bandwidth × RTT)**, and the in-flight limit is constrained by `min(cwnd, receive window, send buffer)`—if any of these three is smaller than the BDP, bandwidth will not be fully utilized.

| Link | BDP | Is Linux Default Sufficient? |
|---|---|---|
| 1Gbps × 30ms (Intra-province, China) | 3.75 MB | Yes (tcp_rmem max default ~6MB) |
| 1Gbps × 150ms (Trans-oceanic) | 18.75 MB | **No** |
| 10Gbps × 200ms | 250 MB | **Far from enough** |

```bash
# Triplet: min / default / max — auto-tuning only scales between min and max
sysctl net.ipv4.tcp_rmem="4096 131072 67108864"   # Receive side, 64MB max
sysctl net.ipv4.tcp_wmem="4096 16384 67108864"    # Send side
sysctl net.core.rmem_max=67108864                  # Upper limit for manual SO_RCVBUF setting
```

Verification method: Use `ss -ti` to check `cwnd`, `rtt`, and `bytes_acked` for a single connection. If `cwnd` stays pinned to the buffer limit rather than fluctuating with the link, the buffer is the bottleneck.

**Note**: Buffers are not "the bigger, the better." Large buffers on low-BDP links just waste memory, and on devices that are themselves bottlenecks, they exacerbate queuing delay. Set them according to the maximum BDP you need to support, leaving a 2x margin.

## Latency Spikes Under Load: Bufferbloat and Queue Management

Classic symptom: **Ping is 10ms when idle, but jumps to 500ms as soon as a download starts**. The cause is not bandwidth, but queues—some device in the path (often a home router or optical modem) is configured with a huge FIFO buffer. Loss-driven algorithms like CUBIC will keep sending packets until it fills up, causing all traffic to queue in this deep buffer.

On the ends you can control, replace FIFO with **AQM (Active Queue Management)**:

```bash
tc qdisc replace dev eth0 root fq_codel              # Default on most distros
tc qdisc replace dev eth0 root cake bandwidth 950mbit # First choice for home egress
```

- **fq_codel** = Fair Queuing + CoDel. CoDel's approach is to monitor the *residence time* of packets: if it consistently exceeds 5ms, it starts dropping packets (from the head of the queue, sending a fast signal) to force senders to slow down—shorter queues mean lower latency. The fair queuing component ensures one elephant flow doesn't starve an adjacent SSH session.
- **cake** is an engineering improvement over fq_codel: it includes a shaper (set `bandwidth` slightly below the real upstream capacity to reclaim queue control from the optical modem), per-host isolation (one device running BT won't affect the whole household), and DiffServ awareness.

The best results are achieved when combined with `bbr` on the sending side—BBR itself does not fill queues (see the congestion control section in [TCP Deep Dive](/networking/03-l4-transport-layer/01-tcp.md)):

```bash
sysctl net.ipv4.tcp_congestion_control=bbr
```

## Common TCP Tuning Knobs

| Knob | Function | When to Adjust | Risk |
|---|---|---|---|
| `tcp_congestion_control=bbr` | Model-driven congestion control | Long-fat/lossy/bufferbloat links | Fairness with CUBIC flows (v1) |
| `tcp_fastopen=3` | Data in SYN, saves 1 RTT for returning clients | Short-lived, high-frequency API services | Middleboxes may drop TFO packets |
| `tcp_tw_reuse=1` | Client-side TIME_WAIT reuse | High-concurrency short-lived connections on the *client* | Safe only for clients; avoid the now-removed tw_recycle |
| `somaxconn` + `tcp_max_syn_backlog` | Full/half connection queue depth | When `ss -lnt` shows listen queue overflow | Only deepens the queue; if processing can't keep up, you still need to scale out |
| `tcp_notsent_lowat=16384` | Limits the amount of "unsent" data in the send buffer | Latency-sensitive streaming applications | Slight reduction in extreme throughput |

Principle: **Test with load after every change**. Knobs interact with each other; changing everything at once is equivalent to changing nothing.

## Small Packets and CPU: NIC Offload

At 10Gbps, packet rates can reach millions per second. Processing interrupts per packet will exhaust CPU before bandwidth is saturated. The solution is **batching**—moving segmentation/merging tasks from the kernel to the NIC:

| Offload | Direction | Function |
|---|---|---|
| TSO | Transmit | Kernel passes a 64KB chunk to the NIC, which hardware-segments it into MSS sizes |
| GSO | Transmit | Software fallback for TSO (virtual NICs, hardware lacking TSO support); segments just before the driver |
| GRO | Receive | Merges consecutive small packets from the same flow into large chunks before passing them up the protocol stack, saving interrupts and traversal overhead |

```bash
ethtool -k eth0 | grep -v fixed     # View current offload status
ethtool -K eth0 tso off             # Temporarily disable when troubleshooting
ethtool -G eth0 rx 4096 tx 4096     # Driver ring buffer; increase when dropping packets during burst traffic
```

**A classic tcpdump trap**: Seeing 64KB "jumbo frames" in captures is not abnormal—it means TSO/GRO is active, and the capture point is before segmentation/after merging; actual transmission on the wire uses MTU-sized packets. Conversely, if you suspect offload is causing issues (especially checksum errors in virtualized environments), first try `ethtool -K ... off` to rule it out.

## Order for Locating Bottlenecks

1. **Baseline**: `iperf3 -c <server> -R` to measure bidirectional bandwidth, getting the raw bandwidth—first confirm whether the bottleneck is indeed in the network.
2. **Measure latency under load**: Run iperf3 and `ping` simultaneously. If latency increases tenfold → bufferbloat, adjust queues.
3. **Inspect single connections**: Check `cwnd`/`rtt`/retrans in `ss -ti`. If `cwnd` cannot increase, check buffers and congestion algorithm; if retransmissions are high, check link quality.
4. **Inspect the system**: `nstat -a | grep -i retrans`, `ethtool -S eth0 | grep -i drop` (NIC layer drops), `softnet_stat` backpressure.
5. Change one variable, then return to step 1.

For systematic packet capture analysis, see [tcpdump and Wireshark](/networking/13-network-diagnostics/01-tcpdump-and-wireshark.md). For troubleshooting methodologies, see [Network Troubleshooting Methodology](/networking/13-network-diagnostics/02-network-troubleshooting-methodology.md).

## References

- **BBR**: github.com/google/bbr · RFC 9438
- **fq_codel/CoDel**: RFC 8290, RFC 8289 · bufferbloat.net
- **cake**: man tc-cake · Engineering follow-up to RFC 8290
- **Brendan Gregg**: "Systems Performance" Chapter 10 (Network)

*Keywords: BDP, BBR, CUBIC, bufferbloat, fq_codel, cake, AQM, TCP tuning, tcp_rmem, TSO, GSO, GRO, ethtool, iperf3*