--- title: Distributed Tracing url: https://doc.liz6.com/en/distributed-systems/08-observability/01-distributed-tracing locale: en area: distributed-systems tags: - distributed-systems - observability date: 2026-06-30 modified: 2026-07-16 description: 'After a request spans over a dozen microservices, how do you know where the time was spent? Distributed tracing assigns a trace ID to each request. Cross-service call relationships pass the span context (trace ID + parent span ID + sampling flag) via context propagation, piecing together fragments scattered across nodes into a complete call chain. Sampling strategies are key to cost control: full sampling is too expensive, fixed-ratio sampling misses tail errors, and tail-based sampling decides which traces to retain after completion.' --- # Distributed Tracing > After a request spans over a dozen microservices, how do you know where the time was spent? Distributed tracing assigns a trace ID to each request. Cross-service call relationships pass the span context (trace ID + parent span ID + sampling flag) via context propagation, piecing together fragments scattered across nodes into a complete call chain. Sampling strategies are key to cost control: full sampling is too expensive, fixed-ratio sampling misses tail errors, and tail-based sampling decides which traces to retain after completion. ## Dapper (Google 2010) → OpenTelemetry Google's Dapper paper inspired the open-source ecosystem: Zipkin (Twitter) → Jaeger (Uber) → OpenTracing + OpenCensus → **OpenTelemetry** (CNCF, current standard). ## Trace and Span Trace and Span Structure: A Trace consists of multiple Spans Trace — The collection of all spans for an end-to-end request Span — A record of an RPC call, containing the following 7 fields trace_id Unique ID for the trace span_id Unique ID for this span parent_id ID of the caller span (root span has no parent) operation "GET /api/users" / "SELECT * FROM users" start/end Timestamps status OK / ERROR (including error details) attributes Custom tags (http.status_code, db.statement, ...) Key structure: parent_id stitches spans scattered across service nodes back into a tree— The root span has no parent; child spans record the caller's span_id. Span Call Hierarchy Tree: Parent-child structure of a /api/order request Trace: user request → /api/order Span A · GET /api/order root span (no parent) Span B · auth-service.ValidateToken Validate user token Span C · order-service.CreateOrder Create order Span D · postgres: INSERT INTO orders Actual database write operation (leaf span) parent_id stitches spans from auth-service / order-service / postgres back into a tree: The root span (A) has no parent; child spans record the caller's span_id, determining the hierarchy. ## Context Propagation Cross-service trace propagation is critical—each service must know "which span of which trace I am": ``` HTTP: traceparent: 00--- W3C Trace Context (standard): traceparent header gRPC: metadata (grpc-trace-bin) Kafka: message headers ``` `trace_flags`: bit 0 = SAMPLED (whether sampled), other bits reserved. Internal implementation: The tracer creates a new span (as a child of the caller span) when calling downstream services, placing `trace_id` + `new_span_id` into outbound headers. Downstream middleware/libraries (HTTP client, gRPC stub, DB driver) extract the trace context when receiving the response, automatically creating the child span. ## Sampling Full collection → overhead is unacceptable (~1μs overhead per span + network/storage costs). Sampling strategies are needed: - **Head-based (probabilistic)**: Decide whether to sample at the start of the trace (random 1/100 or 1/1000). Simple, but may miss traces containing errors. - **Tail-based (intelligent)**: Cache all spans (local buffer), and decide whether to retain them after the trace completes based on the result (ERROR? latency > SLO?). Ensures errors are not lost, but has high memory overhead. OpenTelemetry's `tailsampling` processor implements this strategy. ## Jaeger Deployment (Reference) ```bash # Agent (one per host, receives spans sent by apps via UDP) # Collector (central, receives from agents → writes to storage) # Storage (Cassandra/Elasticsearch) # Query (UI + API) ``` ## References - **Paper**: "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure" (Google, 2010) - **OpenTelemetry**: opentelemetry.io/docs/specs/otel/trace - **Jaeger**: jaegertracing.io *Keywords: distributed tracing, Dapper, OpenTelemetry, Span, trace context propagation, W3C Trace Context, sampling*