--- title: Kafka Architecture url: https://doc.liz6.com/en/distributed-systems/07-messages-and-streams/02-kafka-architecture locale: en area: distributed-systems tags: - distributed-systems - messages-and-streams date: 2026-06-30 modified: 2026-07-16 description: Kafka models the messaging system as an immutable, partitioned, replayable log—not a "consume-and-delete" queue. In-partition ordering guarantees message order for the same key, the ISR mechanism controls the trade-off between replication and durability, and consumer groups turn partition ownership negotiation into a distributed coordination problem. --- # Kafka Architecture > Kafka models the messaging system as an immutable, partitioned, replayable log—not a "consume-and-delete" queue. In-partition ordering guarantees message order for the same key, the ISR mechanism controls the trade-off between replication and durability, and consumer groups turn partition ownership negotiation into a distributed coordination problem. ## Partition: Ordered Immutable Log ``` Topic = multiple partitions: Partition 0: [offset 0][offset 1][offset 2]...[offset N] ← append only Partition 1: [offset 0][offset 1][offset 2]...[offset N] Partition 2: [offset 0][offset 1][offset 2]...[offset N] Producer: choose partition by key: partition = murmur2(key) % N_partitions → same key goes to the same partition → messages for that key are ordered → key=null → round-robin (load balancing, unordered) Consumer: pull-based, offset managed by the consumer ``` ## ISR (In-Sync Replica) Each partition has 1 leader replica + N follower replicas. Only replicas in the ISR are considered "in-sync": ``` Leader: producer writes → leader: append to log → followers: fetch from leader → follower acknowledges: I've replicated up to offset X → if follower doesn't acknowledge within replica.lag.time.max.ms → removed from ISR → min.insync.replicas: each message needs this many ISR replicas to acknowledge to be considered committed ``` If the leader crashes, a new leader is elected from the followers in the ISR. Followers not in the ISR risk having their logs truncated (log diverged from leader). ## Consumer Group A partition of a topic can only be consumed by **one** consumer within the same consumer group. Partitions are distributed among group members. Consumer Group: A partition can only be consumed by 1 consumer in the group Topic: 3 partitions P0 P1 P2 Consumer 1 Consumes P0 Consumer 2 Consumes P1 Consumer 3 Consumes P2 Each partition is consumed by only 1 consumer → message order within that partition is guaranteed Consumer 2 crashes Triggers rebalance Assigned to Consumer 1 or 3 Takes over P2 Rebalance only reassigns partition ownership; at any given time, each partition is still consumed by only 1 consumer, so order guarantees are unaffected Consumer groups guarantee **ordering**: since each partition has only 1 consumer → the messages for that partition seen by the consumer are ordered. ### Offset Commit The consumer manages its own offsets—no broker push is required. Typically, offsets are stored in Kafka's internal `__consumer_offsets` topic: ``` Consumer: poll() → fetch N messages → process → commitSync(offsets) → enable.auto.commit=false: manual commit (recommended, after successful processing) → enable.auto.commit=true: periodic auto-commit (may lose messages) ``` ## Log Compaction Retains the last value based on key, rather than based on time: ``` Log before compaction: key=A, value=X key=B, value=Y key=A, value=Z ← latest value for key A Log after compaction: key=B, value=Y key=A, value=Z → Suitable for: KTable (update stream, latest state) → Not suitable for: KStream (event stream, each change is an independent event) ``` ## References - **Kafka**: kafka.apache.org/documentation/#design - **ISR**: kafka.apache.org/documentation/#replication *Keywords: Kafka, partition, ISR, consumer group, rebalance, offset commit, log compaction*