Distributed Read/Write Path

A cross-node read/write operation where quorum, WAL, Raft logs, and consistent hashing all operate simultaneously. This complete path from coordinator to N replicas and then to read repair is the core data flow of distributed storage systems.

Overview

The previous article on Replication Strategies covered single-leader, multi-leader, and leaderless modes; the Consensus Protocol article explained how Raft enables multiple nodes to agree on a log sequence; and the Consistent Hashing article explained how data is distributed across different nodes. However, during a cross-node read/write operation, these three things happen simultaneously—after a write request lands on a node, how are replicas selected, how is the WAL written, how is quorum waited for, and how is inconsistency repaired on the read path? This article walks through the complete read/write path.

Distributed Write Path: The Journey of a Write Request

Using the leaderless + quorum mode (Dynamo/Cassandra style) as an example, let's trace a write request:

flowchart TD
    C["① Client → Coordinator<br/>Write key=user:42<br/>N=3, W=2"]

    C --> HASH["② Coordinator Consistent Hashing<br/>Determines key falls on 3 nodes<br/>[Node-A, Node-B, Node-C]"]

    HASH --> SEND["③ Coordinator Concurrent Write<br/>→ 3 nodes, waits for W=2 ACKs"]

    SEND --> NODE["On each target node:"]

    NODE --> WAL["a. Append to WAL<br/>→ fsync<br/>Crash recovery guarantee"]
    WAL --> MEM["b. Write to MemTable<br/>(in-memory sorted structure)"]
    MEM --> ACK["c. Return ACK<br/>(containing version/timestamp)"]

    ACK --> QUORUM{"④ Coordinator<br/>Received ≥2 ACKs?"}
    QUORUM -->|"✅ W=2 satisfied"| SUCCESS["Return success to Client"]
    QUORUM -.->|"Not received"| STALE["Remaining nodes' data<br/>temporarily inconsistent"]

    classDef client fill:#e3f2fd,stroke:#1565c0
    classDef coord fill:#fff3e0,stroke:#ef6c00
    classDef node fill:#f3e5f5,stroke:#7b1fa2
    classDef done fill:#e8f5e9,stroke:#2e7d32
    class C client
    class HASH,SEND,QUORUM coord
    class NODE,WAL,MEM,ACK node
    class SUCCESS done

Key: W ≤ N is the tuning knob between "availability" and "consistency." When W=N, the write operation waits for all replicas—if any node fails, the write blocks; when W=1, it is fastest but may read stale data (depending on R). The balance point is usually W + R > N (where R is the read quorum), ensuring that the read and write sets must intersect.

Difference from Raft Write Path

Raft's write is a strong-leader serial process: all writes must go through the leader → leader writes to local log → replicates to follower majority → committed → applied to state machine. There is no quorum tuning knob, and "partial ACKs count as success" does not exist—either it is committed (deterministically persisted) or it is not committed (may be overwritten by a subsequent leader). This is a different world from leaderless quorum: the determinism of Consensus Protocols vs. availability-first Replication Strategies.

Distributed Read Path: How to Read Correctly

In the same scenario, N=3, R=2 (read quorum):

flowchart TD
    C2["① Client → Coordinator<br/>read('user:42')"]

    C2 --> HASH2["② Coordinator Consistent Hashing<br/>Finds N=3 replica nodes"]

    HASH2 --> SEND2["Concurrent read requests, wait for R=2 responses"]

    SEND2 --> COMPARE["③ Received 2 responses<br/>Node-A: version=5<br/>Node-B: version=3"]

    COMPARE --> PICK["Pick latest version=5<br/>Return to Client ✅"]

    COMPARE --> REPAIR["④ Read Repair (asynchronous)<br/>Discovered Node-B is lagging<br/>Background push version=5 → Node-B"]

    classDef client fill:#e3f2fd,stroke:#1565c0
    classDef coord fill:#fff3e0,stroke:#ef6c00
    classDef done fill:#e8f5e9,stroke:#2e7d32
    classDef repair fill:#f3e5f5,stroke:#7b1fa2
    class C2 client
    class HASH2,SEND2,COMPARE coord
    class PICK done
    class REPAIR repair

When R + W > N, read and write quorums must overlap—reads will always see the latest writes. However, there is a nuance: the coordinator must compare versions after receiving at least R responses (not just picking the fastest one), otherwise it might read stale data.

Read Repair vs. Hinted Handoff: Two Repair Mechanisms, Two Timing Points

The core idea of hinted handoff: do not degrade the write; temporarily store the data intended for the downed node elsewhere, and push it later when it recovers. This way, even if some nodes are temporarily unreachable, W can still be achieved (N remains unchanged)—write availability does not drop. Cassandra stores hints for 3 hours by default, discarding them after timeout.

Sloppy Quorum: Pushing Availability to the Extreme

The quorum mentioned above is strict quorum: writes must land on the N "correct" nodes calculated by consistent hashing. If too many of those N nodes fail, even with hinted handoff, W cannot be met.

Sloppy quorum relaxes this: the coordinator can choose any node not in N to satisfy W. The data is temporarily written to a node that "shouldn't have this data," and then moved to the correct nodes after they recover. This further improves write availability, at the cost of reads potentially not seeing the just-written data at all (if the read quorum happens to not cover that "temporary node")—the eventual consistency window may increase significantly.

The Dynamo paper lists both strict and sloppy quorum; Cassandra defaults to strict (with hinted handoff as a fallback); DynamoDB's on-demand mode is closer to sloppy.

From Logs to State Machine: The Dual Role of WAL in Distributed Systems

In Raft, logs are the "carrier of consensus"; in single-node storage engines, WAL is the "carrier of persistence." In actual systems, these two layers interact:

• Raft logs manage which operations have been "determined at the cluster level," and the state machine is replayed and redone after crash recovery by replaying logs.
• Local WAL manages "don't lose the single apply before it is flushed to SSTable"—if a crash occurs after apply but before flush, WAL recovery restores it; after flush, the corresponding part of the WAL can be truncated.
• This is why the Consensus Protocol: Raft article says "committed → apply to state machine" without expanding on what the state machine is—the state machine contains this layer of WAL + storage engine.

Write Amplification: Distributed Systems Make Existing Amplification More Complex

The write amplification in single-node LSM comes from compaction (rewriting layer by layer). Distributed systems add two more layers:

• Network Amplification: When N=3, one write becomes 3 network messages (N replicas). With hinted handoff, it could be even more.
• Raft Replication Amplification: Raft first writes logs (majority), then applies to the state machine (local write WAL + MemTable). One logical write may trigger two local fsyncs (log fsync + WAL fsync)—CockroachDB/Yugabyte's optimization is to merge them into one.

Therefore, the "write throughput" of distributed systems cannot be judged solely by the throughput of the single-node storage engine; network and consensus overheads must be included.

Trade-offs and Failure Modes

• When W + R ≤ N, you cannot read the latest write: Read and write quorums do not guarantee overlap → Always set W + R > N.
• Read repair is not timely: Read discovers but does not repair (forgot to push asynchronously) → Monitor replica version differences, perform periodic anti-entropy (see next article).
• Hinted handoff backlog: If the target node does not recover for a long time, hints will fill up the coordinator's disk → Set hint expiration time, monitor hints queue length.
• fsync policy is too loose: WAL batch fsync causes loss of the last batch during a crash → Set fsync policy according to persistence requirements (Raft logs already have persistence, so local WAL can be relaxed appropriately).

References

• Paper: "Dynamo: Amazon's Highly Available Key-value Store" (DeCandia 2007) — The source of quorum + hinted handoff + sloppy quorum

Keywords: quorum write, quorum read, N/W/R, strict quorum, sloppy quorum, read repair, hinted handoff, coordinator, WAL, fsync, Raft log vs WAL, write amplification, network amplification, consistent hashing + replication, eventual consistency window