Network Performance and Tuning

Network tuning must optimize meaningful workload and user wait times simultaneously, rather than simply inflating a single speed test number. First, define the objective: effective throughput for large files, P99 latency for short requests, connection establishment rate, or packets per second (PPS) for small packets; different objectives may require different batching, queuing, and concurrency strategies.

First, Confirm Which Type of Constraint Is the Bottleneck

Bandwidth underutilization may be caused by single-flow window limits, congestion, application supply, CPU, encryption, storage, rate limiting, or receive processing. A small cwnd does not automatically prove that the socket buffer is small; if the sending application is not continuously supplying data, expanding the window will not magically generate load.

BDP: Estimating Required In-Flight Allowance

Bandwidth-Delay Product BDP = Rate × RTT represents the amount of in-flight data required to cover one round-trip wait time when sending continuously at that rate. It is a starting point for capacity planning, not a directive to pre-allocate memory for every socket.

After unifying cwnd, receive window, and in-flight data into bytes, you can estimate the rate upper bound for a fixed window. For example, with a usable window of 1 MiB and an RTT of 100 ms, even without packet loss, the theoretical upper bound is only about 83.9 Mbit/s; slow start, pacing, and application supply will further affect the result. ss often displays cwnd in segments, which must be combined with MSS and cannot be directly compared to byte-based buffers.

from fractions import Fraction as F
bdp = 1_000_000_000 * F(30, 1000) / 8
assert bdp == 3_750_000
window_limit = 1_048_576 * 8 / F(100, 1000)
assert window_limit == F(83_886_080)
queue_ms = 4_000_000 * 8 / F(100_000_000) * 1000
assert queue_ms == 320
print('BDP 3.75 MB; 1 MiB window upper bound 83.88608 Mbit/s; backlog delay 320 ms')

If you use an RTT that already includes severe queuing to expand buffers, you may mistake queuing for a "pipeline that must continue to be filled." You should combine baseline/minimum RTT with actual contention conditions, rather than always multiplying the maximum RTT by bandwidth.

When Is More In-Flight Data Just Adding to the Queue?

At a default of 1 Gbit/s and 30 ms, 3.75 MB exactly covers the baseline BDP. Continuing to increase the in-flight budget lengthens the golden queue but does not raise the throughput upper bound; reducing it below the BDP causes the queue to disappear and the window to start limiting throughput.

$$BDP=CRT{T}_0,R\le \min (C,W/RT{T}_0),d_q=\max (0,W-BDP)/C$$

Linux Buffers: Upper Limits, Actual Allocation, and Advertised Windows Differ

The three values in tcp_rmem and tcp_wmem describe memory policies for min/default/max; receive auto-tuning is also affected by conditions such as tcp_moderate_rcvbuf. Explicitly setting SO_RCVBUF/SO_SNDBUF in the application changes the corresponding auto-tuning behavior; you cannot assume the kernel still grows strictly according to automatic limits.

# Read current settings only, first confirming the actual values on the current machine and namespace.
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem \
  net.ipv4.tcp_moderate_rcvbuf net.core.rmem_max net.core.wmem_max
ss -tinm

Kernel default values may vary by version and memory size, and distributions may override them. You cannot judge all machines by the statement "Linux default receive max is about 6 MB." Linux 7.2 documentation clearly distinguishes between auto-tuning limits and manual socket options; the auto-maximum of tcp_wmem does not override net.core.wmem_max. Linux IP Sysctl

Linux's internal accounting for SO_RCVBUF/SO_SNDBUF parameters includes extra space; getsockopt often returns doubled values; socket memory usage, data payload, and TCP advertised window should not be treated as the same number. socket(7)

Changing the receive limit to 64 MiB does not mean every connection immediately consumes 64 MiB; however, if 10,000 active connections average 4 MiB actual usage, this memory class alone accounts for approximately 39.1 GiB, not counting other buffers and metadata. Raising the limit should simultaneously evaluate concurrency, memory pressure, and whether the original limit was truly reached.

Queue Management: Only Meaningful When Controlled to the Bottleneck

Queues absorb short bursts but also increase wait times. A 4 MB backlog before a 100 Mbit/s egress link adds approximately 320 ms of serialization wait; no matter how much the queue is expanded, the egress will not become 1 Gbit/s as a result. When sustained input exceeds output, congestion feedback, shaping, or application backpressure is required.

CoDel observes the residence time of packets in the queue and determines whether the minimum residence time remains above the target for a sustained interval, thereby identifying persistent queues; it does not drop any packet immediately just because it exceeds the default 5 ms threshold. Signals can be drops, or ECN marking where applicable. CoDel

FQ-CoDel distributes flows into queues and schedules them, working with CoDel to control persistent waits in each queue. It can improve coexistence between interactive flows and bulk flows, but hash classification, varying flow counts, and implementation conditions mean that "absolute fairness and never starving" is not a universal guarantee. FQ-CoDel

CAKE combines shaping, queue management, flow/host isolation, and DiffServ mechanisms. It is not simply a renamed fq_codel, nor is it necessarily optimal for all interfaces. Shaping rates, link-layer overhead, and actual bottlenecks must match; downstream traffic also requires the ability to implement control at the appropriate ingress/forwarding path. tc-cake

First, observe the existing structure with tc -s qdisc show dev eth0, then decide where to schedule or shape. Replacing the root qdisc may replace existing business policies; record and understand the existing configuration before operating; tuning articles should not directly apply a replace command to all machines.

CUBIC, BBR, and Pacing: Algorithm Names Are Not Performance Conclusions

CUBIC adjusts send permissions based on congestion events and window growth functions; RFC 9438 describes CUBIC, not BBR. BBR builds a path model based on delivery rate, RTT, etc., and controls pacing/inflight; different generations have different behaviors regarding packet loss, ECN, and probing. CUBIC RFC 9438, BBR Official Project

BBR's startup and bandwidth probing can create queues, and it does not guarantee superiority over CUBIC at all times. You must report the exact kernel/patch version, competing flows, RTT, packet loss sources, and measurement objectives. A locally named bbr module does not automatically mean the latest generation from the Google repository is deployed.

sysctl net.ipv4.tcp_available_congestion_control \
  net.ipv4.tcp_congestion_control
tc -s qdisc show dev eth0

Global default congestion algorithms mainly affect subsequent connections of that type; applications may also select algorithms per socket; changing sysctl once does not prove that all existing connections have switched. Send pacing cooperates with qdisc/kernel implementations; fq, fq_codel, and cake are not the same mechanism, and you cannot assume they have identical capabilities just because their names contain 'fq'. tc-fq

Small Packets: Calculate PPS First, Then Look at CPU Distribution

For ideal 10 Gbit/s Ethernet line-rate minimum frames, calculated with 64 B frames (including FCS), 8 B preamble/SFD, and 12 B inter-frame gap, each frame takes 84 B of time, resulting in approximately 14.88 Mpps. This places a completely different load on the CPU compared to "10 Gbit/s large file downloads."

from fractions import Fraction as F
small_pps = F(10_000_000_000, (64 + 8 + 12) * 8)
large_pps = F(10_000_000_000, (1518 + 8 + 12) * 8)
assert 14_880_000 < small_pps < 14_890_000
assert 812_000 < large_pps < 813_000
assert 18 < small_pps / large_pps < 19
assert F(10_000 * 4 * 1024**2, 1024**3) == F(625, 16)
print('Ideal minimum frame 14.88 Mpps, 1518 B frame 0.813 Mpps, average buffer ~39.06 GiB')

This calculation ignores other rate definition differences such as higher-layer physical encoding, and does not imply that the current host can achieve this PPS. At high packet rates, you should not assume that each packet triggers an independent interrupt; NAPI, interrupt coalescing, and batching will change CPU costs.

Viewing load by CPU, IRQ, queue, and NUMA location is usually more meaningful than simply maximizing all rings. Offload issues should also be checked against packet capture locations and driver statistics; disabling offloads changes CPU and throughput, and must be treated as a controlled variable under explicit assumptions. Linux Scaling, Segmentation Offloads

What Evidence Should Common Parameters Correspond To?

Linux 7.2's tcp_tw_reuse distinguishes between 0 (disabled), 1 (globally enabled), and 2 (loopback only); you cannot simply conclude "set to 1, safe for clients." First, check connection reuse, destination distribution, source port constraints, and business lifecycle; deleting large numbers of TIME_WAITs is not an optimization goal. Linux IP Sysctl Parameters

Let Tests Answer Specific Questions

The following commands are for environments with authorized test services; the iperf3 server must be prepared by both parties in advance.

# Client sends to server, single flow.
iperf3 -c perf.example -t 30 -P 1 -J

# Reverse: server sends to client, not simultaneous bidirectional testing.
iperf3 -c perf.example -t 30 -P 1 -R -J

# Controlled multi-flow; do not mix results with single-flow data into one number.
iperf3 -c perf.example -t 30 -P 4 -J

-R only reverses the direction; simultaneous bidirectional is a different test condition. iperf3 measures the throughput of the tool and endpoints under those conditions, not "bare network bandwidth" detached from CPU, kernel, and sockets. UDP test send target rates, receiver goodput, losses, and out-of-order packets should also be reported separately. iperf3 Official Manual

A single control test should record at least the test version, direction, number of flows, duration, packet length or block size, CPU on both ends, throughput, RTT/tail latency under load, and error rate. To avoid misleading results from external load changes, you can alternate running baseline and candidate configurations. If throughput increases by 5%, but P99 doubles or the error rate rises, the result should be evaluated against the original objective, rather than just reporting the fastest single instance.

Specific evidence collection is covered in tcpdump and Wireshark, and phase positioning is covered in Network Troubleshooting Methodology. Each change should correspond to an observed limitation and verify its improvement and cost under the same workload.

Connect measurements to queueing models

When latency rises sharply near the throughput limit, use queueing measurement and simulation to fix the request definition, concurrency model, and observation window. Then use Little’s law to check mean in-flight work, throughput, and mean response time. A relation between means does not determine p99; distinguish slower service, longer waits, and changes in the load generator.