---
title: Service Discovery
url: https://doc.liz6.com/en/distributed-systems/05-members-and-discovery/03-service-discovery
locale: en
area: distributed-systems
tags:
- distributed-systems
- members-and-discovery
date: 2026-06-30
modified: 2026-07-16
description: 'In a dynamically scaling cluster, how do callers know which IP to connect to? Service discovery standardizes the "register → health check → query" process: consul/etcd store the registry, while DNS-SD and client-side load balancing represent two different query paths. Health checks determine "whether this instance can still accept traffic."'
---

# Service Discovery

> In a dynamically scaling cluster, how do callers know which IP to connect to? Service discovery standardizes the "register → health check → query" process: consul/etcd store the registry, while DNS-SD and client-side load balancing represent two different query paths. Health checks determine "whether this instance can still accept traffic."

## Problem

The IP:port of service instances changes dynamically (scaling, rolling updates, failover). How do callers know "which instances are currently available"?

## DNS-SD (Service Discovery via DNS)

Leverage mature DNS infrastructure for service discovery:

```
Service: api-service → DNS SRV records:
  _api._tcp.example.com SRV 10 60 8080 instance-1.example.com
  _api._tcp.example.com SRV 10 40 8080 instance-2.example.com

client: dig SRV _api._tcp.example.com → [instance-1:8080, instance-2:8080]
```

Advantages: No additional components required. Limitations: DNS TTL determines change latency (typically 30-60s), and there is no native health check.

## Consul/etcd Service Registration

Service instance starts → register to Consul/etcd (with health check URL) → client queries → returns healthy instances:

<svg viewBox="0 0 720 340" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,'Source Han Sans CN','Microsoft YaHei',sans-serif" role="img" aria-label="Consul/etcd Service Registration and Discovery: Instance Registration, Health Check, Client Query">
  <defs><marker id="sdah" markerWidth="10" markerHeight="8" refX="8" refY="3" orient="auto"><path d="M0,0 L8,3 L0,6 Z" fill="#475569"/></marker></defs>
  <rect width="720" height="340" fill="#ffffff"/>
  <text x="360" y="30" text-anchor="middle" font-size="17" font-weight="700" fill="#1f2933">Consul/etcd Service Registration and Discovery: Register → Health Check → Client Query</text>

  <rect x="40" y="60" width="170" height="44" rx="8" fill="#e2e8f0"/>
  <text x="125" y="87" text-anchor="middle" font-size="13" font-weight="700" fill="#334155">Instance-1</text>
  <rect x="40" y="124" width="170" height="44" rx="8" fill="#e2e8f0"/>
  <text x="125" y="151" text-anchor="middle" font-size="13" font-weight="700" fill="#334155">Instance-2</text>

  <rect x="460" y="60" width="220" height="46" rx="8" fill="#4f46e5"/>
  <text x="570" y="88" text-anchor="middle" font-size="13" font-weight="700" fill="#ffffff">Consul Server</text>

  <rect x="460" y="136" width="220" height="64" rx="8" fill="#0d9488"/>
  <text x="570" y="162" text-anchor="middle" font-size="13" font-weight="700" fill="#ffffff">Consul Agent</text>
  <text x="570" y="182" text-anchor="middle" font-size="11" fill="#ccfbf1">Health Check GET /health</text>

  <rect x="250" y="238" width="220" height="48" rx="8" fill="#e2e8f0"/>
  <text x="360" y="266" text-anchor="middle" font-size="13" font-weight="700" fill="#334155">Client</text>

  <line x1="210" y1="82" x2="458" y2="83" stroke="#475569" stroke-width="1.6" marker-end="url(#sdah)"/>
  <text x="335" y="70" text-anchor="middle" font-size="11" fill="#475569">register(host:8080)</text>
  <line x1="210" y1="146" x2="458" y2="90" stroke="#475569" stroke-width="1.6" marker-end="url(#sdah)"/>
  <text x="300" y="132" text-anchor="middle" font-size="11" fill="#475569">register(host:8080)</text>

  <line x1="570" y1="106" x2="570" y2="134" stroke="#475569" stroke-width="1.6" marker-end="url(#sdah)"/>

  <line x1="360" y1="236" x2="562" y2="202" stroke="#475569" stroke-width="1.6" marker-end="url(#sdah)"/>
  <text x="465" y="212" text-anchor="middle" font-size="11" fill="#475569">GET /v1/health/service/api</text>
  <text x="465" y="227" text-anchor="middle" font-size="10" fill="#64748b">(Returns list of healthy instances)</text>

  <rect x="40" y="298" width="640" height="32" rx="8" fill="#eef2ff"/>
  <text x="360" y="318" text-anchor="middle" font-size="12" fill="#3730a3">Register only establishes the registration relationship; what truly determines "whether it can be discovered" is the Consul Agent's continuous health checks.</text>
</svg>

Supports: HTTP API, DNS interface (compatible with DNS-SD), long-poll watch (change push).

## Client-side vs Server-side LB

```
Client-side:
  [Client] → Query Service Discovery → Get [I1, I2, I3] → Select one itself → Direct Connect
  Examples: gRPC (with Consul/etcd resolver), Finagle, netflix-eureka

Server-side:
  [Client] → LB (Fixed Address) → LB → [I1, I2, I3]
  Examples: kube-proxy, nginx upstream, HAProxy
```

Comparison: Client-side has one less hop (latency↓), but the client must maintain service discovery logic. Server-side clients do not need to perceive backend changes, but the LB becomes a bottleneck and single point of failure.

## Health Check

- **Passive**: Judged based on failures from actual requests (HTTP 5xx, TCP RST). No additional requests needed, but may be misjudged by occasional transient errors.
- **Active**: Periodic GET /health → Expect 200. Reliable but adds load.
- **TTL-based (Consul)**: Agent periodically reports health status to Consul. If no report is received within the timeout → Marked unhealthy → Removed from discovery.

## References

- **Consul**: consul.io/docs/discovery
- **etcd**: etcd.io/docs
- **gRPC name resolution**: github.com/grpc/grpc/blob/master/doc/naming.md

*Keywords: service discovery, DNS-SD, Consul, etcd, health check, client-side LB, server-side LB*