Consensus in Practice in Real Systems

Consensus algorithms provide a reliable foundation for ordering and committing, but users see the API. Whether the API is linearizable, whether watches can recover, and what lease expiration means still require additional mechanisms. When evaluating a system, start from the actual contracts of the calls, rather than translating "internally uses Raft" directly into "all requests are strongly consistent."

Decompose Services into Responsibilities

Write requests enter the replication log after validation; the protocol determines the prefix before committing; the state machine applies commands and updates queryable data. Persistent logs support fault recovery, indices support efficient lookups, MVCC supports historical versions and snapshots, and watches deliver changes to subscribers. These structures serve different purposes and should not be conflated with the entire consensus state of an application database file.

Similarly, business state changes and external effects are not the same thing: consensus decides "task assigned to A," but it does not automatically stop an old worker process from writing to object storage after it pauses and resumes.

etcd: Compare-and-Read, Revisions, and Watches

etcd v3's standard KV requests provide a strictly serializable contract; Range requests can explicitly opt for the weaker serializable read, allowing stale values to be returned from local state. The option names here can easily be confused with database isolation level terminology, so they should be understood according to the etcd API definitions. etcd 3.6 API Guarantees

Linearizable reads require ensuring that you are not relying on a stale view that has lost authority, and waiting for the state machine to catch up to the required position. The figure below illustrates the general ReadIndex approach; specific preconditions are handled by the consensus implementation.

A revision represents the version sequence of KV state changes; a single modification transaction can allow multiple keys to share the same revision. It is not a Raft term and cannot be simply equated with a log index. Confusing the two can lead to errors in cache recovery or fault diagnosis.

Typical cache synchronization first obtains a snapshot at a specific revision, then establishes a watch from subsequent revisions. The client needs to save its processing progress; if the required history has already been compacted, it must re-read the snapshot. A watch is an interface for continuously receiving events, not a simple long polling query performed periodically, nor does it guarantee that business callbacks will never be retried. etcd API: Watch and Versions

Kubernetes clients usually observe resources through the API server's list/watch interface rather than connecting directly to etcd. Distinguishing application contracts from underlying storage helps in understanding where authentication, resource versions, and caches reside.

Why is the read still waiting after authority is confirmed?

Do not treat a permanently cached "I am the leader" boolean value as evidence for the current read. Below, we separate the valid confirmation of the current term from the local lastApplied: advancing only the application cannot make up for the lack of authority basis; receiving only a majority confirmation cannot make unapplied values appear in the query view. The positions in the diagram are indices on the same log, not KV revisions.

Consul: Member Discovery and Directory Commitment are Different Processes

Consul servers maintain state that requires joint decision-making via Raft, while Gossip is used for member discovery and failure detection. Client agents submit requests to servers, but this does not make them Raft voting members.

It is important to avoid interpreting "Gossip reports a node is reachable" as "all services on that node are ready." Member discovery and specific service checks can influence each other, but they observe different objects; service check results, directory states, and query caches also have their own propagation paths.

Consul reads have different consistency modes; default, consistent, and stale cannot be conflated into a single promise. Whether cross-quorum verification is performed, whether followers are allowed to return data, and behavior during failures are interpreted according to the selected interface and mode. Consensus and Consistency Modes in Consul

Therefore, a directory maintained by consensus may still be readable using stale query modes. Consensus is responsible for state decisions, while the query mode determines which view is allowed for this read.

ZooKeeper: Global Update Order Coexists with Local Reads

ZooKeeper's ZAB provides a foundation for ordered broadcast and recovery of updates; it does not only guarantee "the order of requests from the same client, regardless of different clients." ZooKeeper simultaneously describes guarantees regarding update order and client order, but ordinary reads can be processed locally by the connected server, so different clients may see different progress at the same moment.

The difference from Raft is not "Raft logs are always identical, while ZAB logs allow differences." Normal replication, network latency, and recovery all produce differences in log progress; the key points are committed history and visibility semantics.

ZooKeeper sessions and ephemeral znodes are commonly used for member registration and coordination. A temporary disconnection does not mean the session has expired; clients should distinguish between connection status and session lifecycle. For traditional one-shot watches, you must re-read and re-register after receiving a notification; a single notification cannot serve as a complete event log of all intermediate changes. Other watch modes should be understood according to the specific version API. ZooKeeper Programmer's Guide

Why did I read the third value directly after one notification?

Assume a client reads value 1 corresponding to a version using getData(path, watch=true). The server subsequently changes the value to 2, triggering a NodeDataChanged notification which is consumed by the watch; before the client has a chance to re-read, the value changes to 3. At this point, the client receives a "data changed" notification, and the result of the re-read can directly be 3. You cannot infer that only one write occurred just because a single notification was received.

The following compresses this interleaving into sequential calls. read_watch() represents a read with a watch; the notification queue is deliberately limited to storing event types only. It does not simulate network ordering, session reconnection, or the full ZooKeeper API.

class OneShot:
    def __init__(self, value):
        self.value = value
        self.watched = False
        self.notifications = []

    def read_watch(self):
        self.watched = True
        return self.value

    def set(self, value):
        self.value = value
        if self.watched:
            self.notifications.append("NodeDataChanged")
            self.watched = False

node = OneShot(1)
assert node.read_watch() == 1
node.set(2)
node.set(3)
assert node.notifications == ["NodeDataChanged"]
assert node.notifications.pop(0) == "NodeDataChanged"
assert node.read_watch() == 3  # The intermediate 2 was not delivered as data to the client
node.set(4)
assert node.notifications == ["NodeDataChanged"]
assert node.read_watch() == 4

This determines the consumption method: treat notifications as cache invalidation signals, re-read, and rebuild the current state. If the business requires auditing every change from 1→2→3, you need to save replayable change records; counting one-shot watch callbacks cannot reconstruct history.

When re-reading, use read operations that provide the watch parameter. If you perform a standard read first, and then register via another operation, updates between the two operations may fall into a gap. Even when using the correct read with a watch, it is still possible to skip intermediate values between two reads; it solves the association problem between registration and reading, but does not provide the entire history. The registration lifecycle for persistent watches is different, and you cannot interpret data change notifications as a complete value log based on that. ZooKeeper Watch Semantics and Considerations

Leases and Distributed Locks: Success in the Lock Service Does Not Equal Safety of External Actions

Consider worker process A obtaining a lock, then pausing; the lease expires, and B obtains a new lock and starts working. When A resumes, it may still operate under old assumptions. Simply expiring the record in the lock service cannot prevent A from writing to external resources without checking the lock state.

A common protection is to assign monotonically increasing fencing tokens, which are rejected by the actual resource if they are older than the accepted token. Token comparison must apply to the relevant resource scope, and resource checks and modifications must be atomically linked. If the target API does not support such constraints, you must choose other ways to isolate old executors.

This shows that consensus and leases provide coordination basis, but external systems must still enforce permission boundaries. Obtaining a lock once does not allow unconditional continuation of execution under any pause, timeout, or retry.

Operations Care About Quorum, But Also About Recovery Time

In a basic majority scheme, 3 voters require 2, and 5 require 3. Adding one node does not necessarily improve fault tolerance: 4 nodes require 3, still tolerating only 1 voter failure. You must also consider whether these nodes share racks, power supplies, or network failure domains.

Calculate Based on Failure Domain Distribution, Not Just Total Machine Count

The following calculates based on standard majority for a fixed configuration, assuming one availability zone fails completely while the remaining voters communicate. A, B, and C are availability zones; the quantities in the table count only voters; read-only replicas that do not participate in voting cannot fill the quorum.

def zone_survival(layout):
    if not layout or any(n <= 0 for n in layout.values()):
        raise ValueError("Each listed failure domain must have at least one voter")
    total = sum(layout.values())
    quorum = total // 2 + 1
    return quorum, {
        zone: total - count >= quorum for zone, count in layout.items()
    }

assert zone_survival({"A": 2, "B": 1}) == (2, {"A": False, "B": True})
assert all(zone_survival({"A": 1, "B": 1, "C": 1})[1].values())
assert not zone_survival({"A": 3, "B": 1, "C": 1})[1]["A"]
assert all(zone_survival({"A": 2, "B": 2, "C": 1})[1].values())

For example, with layout A:2, B:2, C:1, if A is lost, only exactly three voters remain; if you then take one B node down for maintenance, only two remain. Fault tolerance budget must deduct currently unavailable nodes, rather than recalculating from "max two failures allowed" for every change. Cross-zone deployment also incorporates some cross-zone round trips into the commit path, so fault-tolerant layouts and write latency must be evaluated together.

The calculations above only indicate that the remaining vote count is sufficient; whether a leader can be recovered, whether disks are available, and whether the network eventually stabilizes still determine whether the service actually progresses. Configuration rules during member changes cannot be replaced by this fixed configuration table.

Quorum survival does not equal latency health. Slow disks, long pauses, and application backlogs can lead to request timeouts, frequent elections, or read waits. You should observe leader changes, persistence latency, replication progress, the gap between committed and applied, and client failure rates, rather than just checking if the process is running.

Backup recovery and member changes must especially adhere to product protocols. Simultaneously restarting multiple old data directories cannot replace a clear cluster recovery process.

Quick Reference by Use Case

Large-capacity storage can also use consensus replication, typically scaling through multiple replication groups. Having more data does not mean "you should switch to Gossip"; Gossip is an information propagation method and cannot replace the commit and read semantics required by users.

References

• etcd 3.6 API Guarantees.
• etcd API.
• Consensus in Consul.
• ZooKeeper Programmer's Guide.

Keywords: etcd, Consul, ZooKeeper, ZAB, ReadIndex, revision, watch, lease, fencing