10 min read #distributed-systems #consensus-protocols
On this page

Consensus in Practice

etcd, Consul, ZooKeeper—three engineering solutions to the same consensus problem in three real-world systems. Raft+boltdb, Raft+Gossip, ZAB: consensus protocols are core but not everything; linearizable reads, leases, watch mechanisms, and membership management each have their own engineering trade-offs.

etcd

etcd is the most widely used Raft implementation (written in Go) and a core dependency of Kubernetes—all K8s objects (Pods, Services, ConfigMaps) are stored in etcd.

Architecture

etcd Architecture: gRPC Requests Written to boltdb via Raft Consensus, MVCC Manages Historical Versions Client gRPC etcd server Raft layer Consensus + log replication Storage boltdb (B-tree, persistent) MVCC Multi-version key-value (historical revisions) Write path: client → Raft propose → commit → apply to boltdb → response; MVCC allows reads to backtrack by version.

Each write request → Raft propose → commit → apply to boltdb → response. Read requests have two modes:

  • Serializable Read: Read directly from local boltdb (bypassing Raft) → may return stale data.
  • Linearizable Read (default): Uses the ReadIndex mechanism—1) Leader records the current commitIndex; 2) Sends heartbeats to a majority of followers to confirm it is still the leader; 3) Waits for the state machine to apply up to ≥ that commitIndex; 4) Performs the read. This ensures reads do not see data from a replaced leader.

Watch

etcd supports long-poll watches: the client specifies a key prefix → the server returns change events. Kubernetes' controller pattern (Informer) is implemented based on etcd watches.

Consul

Consul uses Raft for consistent replication of the service catalog, and Gossip (SWIM) for membership discovery and health check propagation—separating "data that requires consistency" from "data that can be eventually consistent."

Consul: Raft Manages Strongly Consistent Data, Gossip Manages Eventually Consistent Data Consul Agent Raft Consul Server 3-5 nodes, Raft replicates service catalog Strongly consistent (who provides what service) Gossip All Consul Agents Membership discovery, health status propagation Eventually consistent (who is still alive) Design Principle: Service registration requires strong consistency → Raft; membership liveness detection can use eventual consistency → Gossip— separating "data that requires consistency" from "data that can be eventually consistent."

Design principle: Service registration (who provides what service) requires strong consistency → Raft. Membership liveness detection (who is still alive) can use eventual consistency → Gossip.

ZooKeeper (ZAB Protocol)

ZooKeeper's ZAB (ZooKeeper Atomic Broadcast) protocol is 8 years older than Raft and has different design goals:

  • Raft: The log on all servers is identical.
  • ZAB: The leader's log is authoritative; followers may have fewer entries than the leader (but these can be overwritten).

ZAB's "ordering guarantee" is not a global linearizability order like Raft, but rather FIFO client order—multiple requests from the same client maintain their sending order, but there is no ordering guarantee between different clients.

Why ZooKeeper Is Still Alive

ZooKeeper's first-mover advantage and the Java ecosystem have led to many legacy systems relying on it (older versions of Kafka, Hadoop, HBase), but new systems increasingly favor etcd (simpler, Go-native, K8s dependency).

Consensus Use Cases

Consensus is not a silver bullet—it solves the problem of "multiple nodes agreeing on a single value":

Use Consensus:              Do Not Use Consensus:
  - Leader election          - Large data replication (use gossip)
  - Distributed locks        - Non-leader-based replication
  - Configuration center     - Health checks
  - Metadata management      - Log aggregation

Most systems do not need consensus—they need eventually consistent replication (Gossip, async replication) or single master + WAL (most SQL DB master-slave replication).

References

  • etcd: github.com/etcd-io/etcd, raft.github.io
  • Consul: consul.io/docs/architecture
  • ZooKeeper: "ZooKeeper: Wait-free coordination for Internet-scale systems" (2010)

Keywords: etcd, Raft, boltdb, Consul, Gossip, ZooKeeper, ZAB, linearizable read, ReadIndex

Related pages

© 2026 liz's doc