Service Discovery

Service names are stable, but instance addresses change with scaling, restarts, and migrations. Service discovery maintains "which candidate instances are currently available for this logical service," while load balancing selects the connection target from those candidates. Discovering a set of addresses and determining whether they can handle business traffic are two related but distinct responsibilities.

Instance Lifecycle: From Startup to Shutdown

Process liveness only indicates that the program is still running; readiness indicates that it can accept a certain type of business traffic. An instance may be alive but not ready if the service has listened on a port but the cache has not yet loaded, or if database migrations are not yet complete. Shutdown should also stop new traffic first, then allow in-flight requests and connections time to drain.

Addresses in the registry may still be stale. Callers need connection failure handling, retry budgets, and update mechanisms; they cannot treat healthy returned by the registry as proof that the current request will definitely succeed.

DNS and DNS-SD

Standard A/AAAA queries yield addresses, while SRV records can express target host, port, priority, and weight. DNS-SD also uses PTR records to browse service instances and retrieves instance location and attributes via SRV/TXT; it is not equivalent to the narrow definition of "just writing a few SRV records." RFC 6763 provides the record organization method.

_api._tcp.example.com. 30 IN SRV 10 60 8080 api-1.example.com.
_api._tcp.example.com. 30 IN SRV 10 40 8080 api-2.example.com.

This is an example of using SRV to express service endpoints. Clients must support the corresponding queries and weight semantics; standard HTTP clients may not automatically query SRV.

TTL limits the validity period of DNS caches, but actual traffic switching is also affected by client caches, re-resolution timing, connection pools, and long-lived connections. A TTL of 30 seconds does not guarantee that no traffic will flow 30 seconds after an instance is removed. The DNS protocol itself does not define business health checks; health status requires additional controllers to update records.

Registry: Differences Between Storage Primitives and Complete Products

Consul provides service registration and health-related functions; etcd provides consistency primitives such as key-value storage, transactions, Lease, and Watch. Applications need to implement service registration conventions and health policies on top of these. It is incorrect to describe both as "automatically checking health when a health URL is submitted."

One etcd approach binds instance keys to a lease. Active instances periodically renew the lease, and the key is removed when the lease expires. Watch allows callers to receive changes; initial reads and subsequent watches must connect to the correct revision to avoid missing updates between the two. When history has been compacted, a new snapshot must be obtained and listening continued. Interface guarantees are described in the etcd API.

Lease timeout only indicates that renewal was not timely, which may be due to network isolation; it still does not prove that the old instance has stopped execution. If the discovery mechanism also handles exclusive task assignment, fencing must be additionally considered.

Snapshots and Watch: Connecting via Revision, Not "Start Listening ASAP"

Suppose a full read is performed on a service prefix, yielding {A,B} at revision 20. Between the read returning and the Watch being established, A is deleted at 21 and C is added at 22. If the Watch starts from "now" when it is established, only subsequent events are received, and the local cache may forever retain A and miss C.

The Watch should start from position 21, the next position after the snapshot revision, allowing the server to replay intermediate history before continuing to receive new changes; if 21 has been compacted, a new full snapshot must be obtained and connected. Compaction should not be treated as "no changes." The starting revision and compaction responses for etcd are described in the Watch API.

The following simulates cache updates. Events are grouped by revision, and apply_revision is only called when all related events for the same revision have arrived. Here, put/delete represent instance addition and removal, without connecting to etcd.

class DiscoveryCache:
    def __init__(self, revision, snapshot):
        self.revision = revision
        self.endpoints = dict(snapshot)

    def apply_revision(self, revision, events):
        if revision <= self.revision:
            return False  # History replay already fully applied
        updated = dict(self.endpoints)
        for operation, instance, address in events:
            if operation == "put":
                updated[instance] = address
            elif operation == "delete":
                updated.pop(instance, None)
            else:
                raise ValueError("Unknown event type")
        # Model called serially by a single thread; only after publishing the entire new view is the full application progress advanced.
        self.endpoints, self.revision = updated, revision
        return True

cache = DiscoveryCache(20, {"A": "addr-A", "B": "addr-B"})
assert cache.apply_revision(21, [("delete", "A", None)])
assert cache.apply_revision(22, [("put", "C", "addr-C")])
assert cache.endpoints == {"B": "addr-B", "C": "addr-C"}
# Revision 23 modified other prefixes; the current Watch has no corresponding events, which is a valid gap.
change = [("delete", "B", None), ("put", "D", "addr-D")]
assert cache.apply_revision(24, change)
assert not cache.apply_revision(24, change)
assert cache.endpoints == {"C": "addr-C", "D": "addr-D"}
assert cache.revision == 24

# If event type is incorrect, do not publish partial views, nor advance the recovery position.
try:
    cache.apply_revision(25, [("delete", "C", None), ("invalid", "E", "addr-E")])
except ValueError:
    pass
assert cache.endpoints == {"C": "addr-C", "D": "addr-D"}
assert cache.revision == 24

# Incorrect usage: Updating the checkpoint line-by-line for the same revision treats the second line as a duplicate.
bad = DiscoveryCache(22, {"B": "addr-B", "C": "addr-C"})
bad.apply_revision(24, [change[0]])
assert not bad.apply_revision(24, [change[1]])
assert "D" not in bad.endpoints

This implementation relies on the upstream delivering complete, ordered events, and cannot independently prove that no events for this prefix were missed between 22 and 24. Revisions do not need to be consecutive integers because writes to other keys also advance the cluster revision. Actual clients must adhere to Watch history and ordering guarantees, handling disconnections, cancellations, and compaction errors. See etcd Watch Guarantees.

A single transaction can modify multiple instance keys, all with the same revision. The recovery checkpoint must only be advanced after all relevant events for that revision have been applied; it cannot persist "24 processed" after the first event. If response sharding is enabled, the complete content must first be reassembled. The modification revision of events, the revision in the response headers, and progress notifications cannot be indiscriminately treated as "the position the local cache has caught up to."

Furthermore, if a large prefix requires paginated reads, the same snapshot revision must be fixed; otherwise, one page comes from 20 and another from 22, meaning the initial set is not even a consistent view. If the cache and recovery checkpoint are to be persisted, they must be persisted together, or the snapshot must be re-fetched upon process restart to avoid the progress being more up-to-date than the actually saved cache.

After the Snapshot, Where Does Listening Resume?

When comparing caches, do not just look at the final address list, but also at which snapshot and which history segment it is composed of. The following allows independently revoking A and registering C after the snapshot, then choosing whether to replay from the next revision of the old snapshot or only listen from now. History compaction will cause the former approach to fail explicitly; the correct recovery is to rebuild the snapshot, not to keep the old list and swallow errors.

Client Selection and Proxy Selection

Proxies can be deployed in multiple replicas, so they are not inherently single points of failure; client direct connection does not mean there is no control plane dependency. gRPC separates name resolution from load balancing strategies; the resolver can provide address changes and service configuration. See Custom Name Resolution.

Long-lived connections are particularly important: many requests in HTTP/2 or gRPC reuse the same connection, and schemes that only select the backend when establishing the connection may result in uneven distribution. Discovering new instances does not automatically migrate requests on existing connections to them.

Deleting an Endpoint Only Changes the Candidate Set, It Does Not Reclaim Existing Connections

Suppose instance A deregisters at t=0, and the client receives the change at t=2, but its connection pool to A still has one available long-lived connection. If the client only queries the candidate set when creating new connections, requests at t=10 may still go through the old connection; DNS TTL or Watch updates alone cannot prevent this.

Therefore, graceful shutdown should simultaneously agree on discovery propagation, connection draining, and task deadlines. If existing requests have already produced business effects but the response is interrupted, retries still need to be idempotent; "graceful" does not guarantee that all requests are completed without loss. The discovery layer provides location changes, while the connection and request layers are responsible for implementing them as actual traffic changes.

How to Combine Health Signals

Active checks perform periodic probes, which can detect problems when there is no business traffic; passive checks observe real requests, which are closer to the user path; TTL reports rely on the checked party actively renewing. All three can produce false positives/negatives and usually require consecutive failure/success thresholds and jitter suppression.

The scope of checks should also be restrained. If a shared downstream brief failure causes all instances to be marked dead, load balancing may remove all candidates; if the health endpoint always returns 200, it cannot discover instances that truly cannot serve. Readiness should be defined according to service degradation capabilities, rather than mechanically checking all dependencies.

What to Do When the Registry is Unreachable

Existing endpoint caches allow the data plane to continue working temporarily, but the longer the cache persists, the staler it becomes. New instances may not be able to join, old instances may have disappeared, and security or policy revocations may not yet have taken effect. Choose the allowed degree of staleness based on business needs: continue using the last known set, only remove consecutively failed endpoints, or stop service in certain permission-sensitive scenarios.

This reflects the separation of control plane and data plane: a registry failure does not necessarily immediately interrupt existing connections, but the longer it lasts, the greater the risk caused by instance changes. When verifying the discovery system, observe propagation delay, connection updates, and actual request success rates simultaneously.

References

• RFC 6763: DNS-Based Service Discovery
• etcd: API, Lease, and Watch
• gRPC: Custom Name Resolution
• Failure Detection

Keywords: service discovery, DNS-SD, SRV, Lease, Watch, readiness, load balancing, connection pool, control plane, data plane