6 min read #distributed-systems #members-and-discovery
On this page

Service Discovery

In a dynamically scaling cluster, how do callers know which IP to connect to? Service discovery standardizes the "register → health check → query" process: consul/etcd store the registry, while DNS-SD and client-side load balancing represent two different query paths. Health checks determine "whether this instance can still accept traffic."

Problem

The IP:port of service instances changes dynamically (scaling, rolling updates, failover). How do callers know "which instances are currently available"?

DNS-SD (Service Discovery via DNS)

Leverage mature DNS infrastructure for service discovery:

Service: api-service → DNS SRV records:
  _api._tcp.example.com SRV 10 60 8080 instance-1.example.com
  _api._tcp.example.com SRV 10 40 8080 instance-2.example.com

client: dig SRV _api._tcp.example.com → [instance-1:8080, instance-2:8080]

Advantages: No additional components required. Limitations: DNS TTL determines change latency (typically 30-60s), and there is no native health check.

Consul/etcd Service Registration

Service instance starts → register to Consul/etcd (with health check URL) → client queries → returns healthy instances:

Consul/etcd Service Registration and Discovery: Register → Health Check → Client Query Instance-1 Instance-2 Consul Server Consul Agent Health Check GET /health Client register(host:8080) register(host:8080) GET /v1/health/service/api (Returns list of healthy instances) Register only establishes the registration relationship; what truly determines "whether it can be discovered" is the Consul Agent's continuous health checks.

Supports: HTTP API, DNS interface (compatible with DNS-SD), long-poll watch (change push).

Client-side vs Server-side LB

Client-side:
  [Client] → Query Service Discovery → Get [I1, I2, I3] → Select one itself → Direct Connect
  Examples: gRPC (with Consul/etcd resolver), Finagle, netflix-eureka

Server-side:
  [Client] → LB (Fixed Address) → LB → [I1, I2, I3]
  Examples: kube-proxy, nginx upstream, HAProxy

Comparison: Client-side has one less hop (latency↓), but the client must maintain service discovery logic. Server-side clients do not need to perceive backend changes, but the LB becomes a bottleneck and single point of failure.

Health Check

  • Passive: Judged based on failures from actual requests (HTTP 5xx, TCP RST). No additional requests needed, but may be misjudged by occasional transient errors.
  • Active: Periodic GET /health → Expect 200. Reliable but adds load.
  • TTL-based (Consul): Agent periodically reports health status to Consul. If no report is received within the timeout → Marked unhealthy → Removed from discovery.

References

  • Consul: consul.io/docs/discovery
  • etcd: etcd.io/docs
  • gRPC name resolution: github.com/grpc/grpc/blob/master/doc/naming.md

Keywords: service discovery, DNS-SD, Consul, etcd, health check, client-side LB, server-side LB

Related pages

© 2026 liz's doc