Measurement, load testing and discrete-event simulation

Measurement and reproducible experiments determine whether a formula fits. A useful load test records intended arrivals, the location of waiting, and unfinished requests, alongside completed work.

Open and closed workloads

A fixed population that submits again after completion produces a closed workload. An open model schedules arrivals independently of response slowdown. Both are legitimate when they match actual demand. k6's official documentation distinguishes these execution models.

Consider a planned request every 10 ms during a 100 ms server pause. A generator waiting for each response before submitting another does not record the requests that would have arrived during the pause: a typical source of coordinated omission. Measuring only actual-send-to-completion can also omit generator delay between intended and actual submission.

Preserve planned arrival, actual send, admission, service start and final outcome. An open generator can exhaust virtual users, CPU, connections or network capacity; report dropped or late iterations and actual arrival rates. HdrHistogram can correct omitted observations using an expected interval, but that correction assumes a workload model and cannot recover every fact about arbitrary real traffic.

Discrete events do not need animation frames

A single FIFO server obeys Lindley's recursion:

$$W_{q,i+1}=\max (0,W_{q,i}+S_i-I_{i+1}).$$

Equivalently, $b_i=\max (a_i,d_{i-1})$ and $d_i=b_i+S_i$. Jump directly between events instead of looping every millisecond, and define a deterministic ordering for ties. SimPy's scheduling guide explains simulation time and event order. This experiment independently implements the recurrence and does not depend on rendering speed.

# Independently generated sequences, using the same time unit.
end = 0.0
samples = []
for arrival, service_time in zip(arrivals, service):
    start = max(arrival, end)
    end = start + service_time
    samples.append((arrival, start, end))

Warmup, censoring and arrival cohorts

An initially empty queue can give early jobs unusually short waits. The experiment discards the first 20% of arrivals as warmup, a declared rule rather than proof of equilibrium at high utilization. Its cutoff is the last arrival time; service then continues until every admitted job finishes.

Report both the full post-warmup arrival-cohort mean and the mean for those completed by the cutoff. The latter excludes unfinished jobs and may underestimate delay, though the difference in one run depends on the trace. In FIFO, stopping future arrivals does not change completion times of already queued jobs, so draining recovers this cohort's complete times. That argument does not transfer to scheduling whose choices depend on future arrivals.

Keep the seed fixed while changing load, then change the seed to see sampling variation. The empirical curve shows exceedance fractions at a finite grid of thresholds; the reference is the exact stationary M/M/1 tail. Sample p99 is the sorted observation at rank $\lceil 0.99n\rceil$. One simulation need not equal theory, and agreement does not prove the model describes a real service.

Results that support decisions

Use independent repetitions and record parameters, seeds, warmup, observation length and uncertainty. Waiting times are often serially correlated: treating one long sequence as IID can produce unjustifiably narrow confidence intervals. Independent replications or justified batch-means methods can help. Near saturation, correlation and recovery times can be long.

Retain successes, rejections, timeouts, cancellations, unfinished counts and class-specific distributions. A timeout says an outcome was not observed by a threshold; it is not an exact completion time. Do not average instance p99 values. Merge compatible histograms or raw samples before computing a combined quantile, stating bucket precision and population definitions.

Check your understanding

1. Roughly how many upper-tail observations support p99 from 1000 samples?

2. Does identical p99 from two runs with the same seed prove correctness?