9 min read #distributed-systems #observability
On this page

Distributed Tracing

After a request spans over a dozen microservices, how do you know where the time was spent? Distributed tracing assigns a trace ID to each request. Cross-service call relationships pass the span context (trace ID + parent span ID + sampling flag) via context propagation, piecing together fragments scattered across nodes into a complete call chain. Sampling strategies are key to cost control: full sampling is too expensive, fixed-ratio sampling misses tail errors, and tail-based sampling decides which traces to retain after completion.

Dapper (Google 2010) → OpenTelemetry

Google's Dapper paper inspired the open-source ecosystem: Zipkin (Twitter) → Jaeger (Uber) → OpenTracing + OpenCensus → OpenTelemetry (CNCF, current standard).

Trace and Span

Trace and Span Structure: A Trace consists of multiple Spans Trace — The collection of all spans for an end-to-end request Span — A record of an RPC call, containing the following 7 fields trace_id Unique ID for the trace span_id Unique ID for this span parent_id ID of the caller span (root span has no parent) operation "GET /api/users" / "SELECT * FROM users" start/end Timestamps status OK / ERROR (including error details) attributes Custom tags (http.status_code, db.statement, ...) Key structure: parent_id stitches spans scattered across service nodes back into a tree— The root span has no parent; child spans record the caller's span_id. Span Call Hierarchy Tree: Parent-child structure of a /api/order request Trace: user request → /api/order Span A · GET /api/order root span (no parent) Span B · auth-service.ValidateToken Validate user token Span C · order-service.CreateOrder Create order Span D · postgres: INSERT INTO orders Actual database write operation (leaf span) parent_id stitches spans from auth-service / order-service / postgres back into a tree: The root span (A) has no parent; child spans record the caller's span_id, determining the hierarchy.

Context Propagation

Cross-service trace propagation is critical—each service must know "which span of which trace I am":

HTTP: traceparent: 00-<trace_id>-<span_id>-<trace_flags>
W3C Trace Context (standard): traceparent header
gRPC: metadata (grpc-trace-bin)
Kafka: message headers

trace_flags: bit 0 = SAMPLED (whether sampled), other bits reserved.

Internal implementation: The tracer creates a new span (as a child of the caller span) when calling downstream services, placing trace_id + new_span_id into outbound headers. Downstream middleware/libraries (HTTP client, gRPC stub, DB driver) extract the trace context when receiving the response, automatically creating the child span.

Sampling

Full collection → overhead is unacceptable (~1μs overhead per span + network/storage costs). Sampling strategies are needed:

  • Head-based (probabilistic): Decide whether to sample at the start of the trace (random 1/100 or 1/1000). Simple, but may miss traces containing errors.
  • Tail-based (intelligent): Cache all spans (local buffer), and decide whether to retain them after the trace completes based on the result (ERROR? latency > SLO?). Ensures errors are not lost, but has high memory overhead. OpenTelemetry's tailsampling processor implements this strategy.

Jaeger Deployment (Reference)

# Agent (one per host, receives spans sent by apps via UDP)
# Collector (central, receives from agents → writes to storage)
# Storage (Cassandra/Elasticsearch)
# Query (UI + API)

References

  • Paper: "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure" (Google, 2010)
  • OpenTelemetry: opentelemetry.io/docs/specs/otel/trace
  • Jaeger: jaegertracing.io

Keywords: distributed tracing, Dapper, OpenTelemetry, Span, trace context propagation, W3C Trace Context, sampling

Related pages

© 2026 liz's doc