Distributed Tracing

When a request passes through a gateway, order service, inventory service, and database, each node's logs only record local facts. Distributed tracing correlates these local operations by causality, answering "where did this request go, and where was the time spent."

It complements metrics and logs: metrics describe the distribution of a large number of requests, tracing shows the structure of a single request, and logs preserve local event details. A single trace can explain a slow request but cannot alone represent the overall failure rate. See Metrics and Alerts for overall measurements.

Trace and Span: Time Intervals with Causality

A trace is a collection of related operations; a span is one such operation with a start and end time, such as a server-side request, a database access, or an internal computation phase. A span typically contains a name, ID, parent span, attributes, events, and status.

The parent-child relationship indicates "in whose context this operation was initiated," which is not equal to "the parent span's duration is the sum of all child spans' durations." Multiple child operations can occur concurrently, and the parent operation may also have uninstrumented computation, queuing, and serialization time.

In the diagram, inventory and price start simultaneously. The end-to-end time depends on the slower price branch, plus the preceding and following stages. Adding all span durations would double-count the overlapping time. The so-called critical path is the chain of dependencies that limits the overall completion time; reducing work not on the critical path may not reduce the latency of this request.

Further distinguish between wall time and CPU time: an RPC span lasts 100 ms, but most of that time might be spent waiting for the network or thread scheduling. Tracing can narrow the scope, but CPU hotspots usually require sampling profilers to confirm.

Calculate Potential Improvements from Waterfall Charts, Not by Adding Span Durations

Using the dependencies in the diagram: after 10 ms of preparation, inventory and price RPCs are initiated in parallel, and after both complete, 10 ms is spent assembling the response. Assuming that shortening one branch does not change other stages, you can calculate the impact of each change on the end-to-end time.

def latency(stock, price):
    return 10 + max(stock, price) + 10

assert latency(70, 100) == 120
assert latency(30, 100) == 120  # Inventory saves 40 ms, but the overall time still waits for Price
assert latency(70, 80) == 100
assert latency(70, 50) == 90    # The critical path has shifted to Inventory

Reducing the price from 100 ms to 50 ms only reduces the overall time from 120 ms to 90 ms, because inventory starts to limit the completion time. This deduction relies on the known business structure that "both RPCs must complete"; if the logic were to take the first success, timeout degradation, or cancel other branches, the formula would change, and you cannot infer the waiting semantics just from two overlapping time bars.

Subtracting child duration from parent duration also requires deduplication. The root interval is [0,120), inventory is [10,80), price is [10,110). The total length of the two child intervals is 170, but the union of the covered time is only 100; the intervals not covered by these two RPCs sum to 20.

def covered(intervals, parent):
    lo, hi = parent
    assert lo <= hi
    clipped = []
    for start, end in intervals:
        if start > end:
            raise ValueError("reversed interval")
        start, end = max(start, lo), min(end, hi)
        if start < end:
            clipped.append((start, end))
    total, right = 0, lo
    for start, end in sorted(clipped):
        total += max(0, end - max(start, right))
        right = max(right, end)
    return total

assert covered([(10, 80), (10, 110)], (0, 120)) == 100
assert covered([(10, 110), (10, 110)], (0, 120)) == 100
assert covered([(-5, 10), (110, 140)], (0, 120)) == 20
assert covered([], (0, 120)) == 0

The 20 ms is merely "wall-clock time not covered by the selected child intervals," which may include queuing, computation, serialization, or instrumentation gaps; it cannot be directly named as the parent service's CPU time. This interval model also assumes all time is on the same comparable timeline; when raw timestamps across machines have offsets, clock issues must be handled first, so that clipping does not secretly mask data anomalies.

How Much Faster Can the Whole Request Be If One Branch Is Shortened?

The following diagram has the two RPCs share the start point after preparation ends, and has the assembly phase follow the later end point. Reducing the price from 100 to 70 ms still shortens the total time; reducing it further, the inventory branch starts to determine the total duration. The path currently limiting completion is highlighted in gold; when both are of equal length, both branches are critical; this explains optimization boundaries better than just marking the longest span.

$$T=T_{\mathrm{prepare}}+\max (T_A,T_B)+T_{\mathrm{finalize}}$$

Context Propagation: How Distributed Calls Are Connected

In a single process, execution context can be passed via thread-local or async task mechanisms; across processes, it must be explicitly encoded into the communication carrier. OpenTelemetry calls writing to the carrier by the sender inject, and reading by the receiver extract.

The correct sequence is: A creates a client span and writes its context into the request; B extracts the context upon receiving the request and creates a server span; when B calls the next service, it creates a new client span and injects it into the next request. It is not that the downstream establishes the parent relationship only after receiving the response. OpenTelemetry Context Propagation explains the relationship between propagators and execution context.

The version 00 structure of the W3C traceparent is as follows, which is a protocol format, not a flowchart:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

tracestate can carry vendor-specific tracing state. gRPC metadata and message headers can also carry propagation fields; do not treat a specific historical binary header as the only format for all gRPC tracing. Field validation and version behavior are governed by W3C Trace Context.

Async Tasks and Messages: The Parent-Child Tree Is Sometimes Insufficient

Synchronous RPCs usually have a clear single caller; message batching may funnel many sources into a single operation. A span can have only one parent, but it can associate multiple span contexts using links. Forcing the sources of a hundred messages into a single-parent chain loses the true relationship of the batch processing.

You must also distinguish between "message waiting in the queue" and "consumer actually processing." The time between the producer finishing the send and the consumer starting may belong to queue residence; when the processing span created by the consumer only covers the execution phase, you cannot treat its duration as end-to-end message latency.

Thread pools are another common boundary: if context is not captured and restored after thread switching, subsequent spans will become new roots; if old context is incorrectly reused, unrelated requests may be linked together. Async propagation must be handled according to the task lifecycle, not just the thread lifecycle.

See OpenTelemetry Traces for the relevant data model.

Sampling: The Trade-off Between Cost and Interpretability

Head sampling decides whether to record at an earlier stage, facilitating control over SDK and transmission costs. Probabilistic sampling is suitable for obtaining an approximately representative sample, but low-frequency errors might coincidentally all be missed. Associated services should maintain compatible sampling decisions, otherwise, disconnected traces are likely.

Tail sampling collects spans and then decides which traces to keep based on errors, latency, or attributes. The cost is receiving data first, maintaining caches, and waiting. Spans of the same trace need to be routed to a location where a joint decision can be made; late-arriving spans, extremely long tasks, and insufficient capacity all affect completeness. OpenTelemetry Sampling explains the capabilities and boundaries of both.

A custom capacity estimate: assume 2000 traces per second, with an average of 8 spans per trace, each span cache occupying 1 KB, and a wait time of 15 seconds. Then the raw capacity for just these spans is approximately 2000 × 8 × 1 KB × 15 = 240 MB. In practice, you must also add index, object, and queue overhead. This estimate is for understanding growth direction, not as a product performance guarantee.

Tail sampling keeping all observed errors does not mean it kept all errors that occurred in the system; the upstream might not have instrumentation, might not have exported, or might have been discarded by head sampling. You also cannot directly calculate the total error rate by "number of error traces / number of kept traces," because the sample is already biased towards anomalous requests.

Why Does the Error Proportion in the Sample Change from 0.1% to About 9.1%?

Assume a batch of 100,000 requests has 100 errors, with a true error rate of 0.1%. Now keep all errors, but keep only 1% of normal requests; if this batch happens to leave 999 normal traces, the error proportion in the sample is 100/1099 ≈ 9.10%. The two proportions describe different populations and cannot replace each other.

from fractions import Fraction
all_errors, all_successes = 100, 99900
kept_errors, kept_successes = 100, 999
true_rate = Fraction(all_errors, all_errors + all_successes)
sample_rate = Fraction(kept_errors, kept_errors + kept_successes)
assert true_rate == Fraction(1, 1000)
assert Fraction(9, 100) < sample_rate < Fraction(10, 100)
# Estimate by weight only in a model where the actual inclusion probability of each type is known and there is no other loss.
estimated_errors = Fraction(kept_errors, 1)
estimated_successes = Fraction(kept_successes, 1) / Fraction(1, 100)
assert estimated_errors / (estimated_errors + estimated_successes) == true_rate

The weight formula is not a universal fix for arbitrary tail sampling data. In practice, data might also be discarded based on latency, attributes, or capacity policies, and routing and export failures will also change inclusion probabilities; unknown losses cannot be automatically recovered from the traces remaining at the end. Even if probabilities are known, the proportion estimate of a finite random sample has variance; this example is just a case that happens to take the corresponding quantity.

Therefore, error rates and SLOs should usually be calculated by semantically consistent request metrics, with traces used to explain why the selected requests failed. If metrics are derived from spans, you must clarify whether they are generated before or after sampling, otherwise, they may still inherit the same selection bias. OpenTelemetry Sampling

Collection Pipeline and Responsibilities

A common pipeline consists of application instrumentation, SDK, Collector, and storage/query backend. Instrumentation is responsible for selecting operation boundaries; the SDK manages span lifecycle and export; the Collector can handle batching, filtering, sampling, and routing; the backend is responsible for retention and retrieval.

Automatic instrumentation usually provides HTTP, RPC, and database boundaries first, while manual spans are used to supplement business-meaningful stages. Adding spans to every tiny function increases overhead and makes the call graph hard to read. Span names should be stable, such as the route template /orders/{id}, with specific request IDs placed in controlled attributes.

Baggage is an application key-value pair propagated with the context; it is not a span attribute, nor is it an authenticated identity claim. Entry points accepting external context need validation and trust boundaries; do not put secrets or full personal information into fields that will be propagated across services. See OpenTelemetry Baggage.

How to Read a Slow Request

First, confirm whether the trace is complete, then look for the critical path in the root span's total duration. If the parent span is long but all child spans are short, check for uncovered queuing, application computation, and serialization stages; if a client span is long but the server span is short, consider connection establishment, network, proxy, or waiting outside of server-side instrumentation.

Cross-machine clocks may have deviations; you cannot assert precise network latency based solely on the absolute timestamps of two nodes. The trace's parent/link relationships provide causal clues, and the timeline provides observations; the two should not be mixed into a perfect global clock.

Finally, verify whether this phenomenon is common using metrics from similar requests, then jump to logs or profiling data via the trace ID. The role of tracing is to propose and narrow down explanations, not to let a single waterfall chart replace all evidence.

References

• W3C: Trace Context
• OpenTelemetry: Context Propagation
• OpenTelemetry: Traces
• OpenTelemetry: Sampling
• OpenTelemetry: Baggage

Keywords: trace, span, context propagation, critical path, links, head sampling, tail sampling, baggage