5 min read #distributed-systems #Messages and Streams
On this page

Kafka Architecture

Kafka models the messaging system as an immutable, partitioned, replayable log—not a "consume-and-delete" queue. In-partition ordering guarantees message order for the same key, the ISR mechanism controls the trade-off between replication and durability, and consumer groups turn partition ownership negotiation into a distributed coordination problem.

Partition: Ordered Immutable Log

Topic = multiple partitions:
  Partition 0: [offset 0][offset 1][offset 2]...[offset N]  ← append only
  Partition 1: [offset 0][offset 1][offset 2]...[offset N]
  Partition 2: [offset 0][offset 1][offset 2]...[offset N]

Producer: choose partition by key: partition = murmur2(key) % N_partitions
  → same key goes to the same partition → messages for that key are ordered
  → key=null → round-robin (load balancing, unordered)

Consumer: pull-based, offset managed by the consumer

ISR (In-Sync Replica)

Each partition has 1 leader replica + N follower replicas. Only replicas in the ISR are considered "in-sync":

Leader: producer writes → leader: append to log → followers: fetch from leader
  → follower acknowledges: I've replicated up to offset X
  → if follower doesn't acknowledge within replica.lag.time.max.ms → removed from ISR
  → min.insync.replicas: each message needs this many ISR replicas to acknowledge to be considered committed

If the leader crashes, a new leader is elected from the followers in the ISR. Followers not in the ISR risk having their logs truncated (log diverged from leader).

Consumer Group

A partition of a topic can only be consumed by one consumer within the same consumer group. Partitions are distributed among group members.

Consumer Group: A partition can only be consumed by 1 consumer in the group Topic: 3 partitions P0 P1 P2 Consumer 1 Consumes P0 Consumer 2 Consumes P1 Consumer 3 Consumes P2 Each partition is consumed by only 1 consumer → message order within that partition is guaranteed Consumer 2 crashes Triggers rebalance Assigned to Consumer 1 or 3 Takes over P2 Rebalance only reassigns partition ownership; at any given time, each partition is still consumed by only 1 consumer, so order guarantees are unaffected

Consumer groups guarantee ordering: since each partition has only 1 consumer → the messages for that partition seen by the consumer are ordered.

Offset Commit

The consumer manages its own offsets—no broker push is required. Typically, offsets are stored in Kafka's internal __consumer_offsets topic:

Consumer: poll() → fetch N messages → process → commitSync(offsets)
  → enable.auto.commit=false: manual commit (recommended, after successful processing)
  → enable.auto.commit=true: periodic auto-commit (may lose messages)

Log Compaction

Retains the last value based on key, rather than based on time:

Log before compaction:
  key=A, value=X
  key=B, value=Y
  key=A, value=Z    ← latest value for key A

Log after compaction:
  key=B, value=Y
  key=A, value=Z

→ Suitable for: KTable (update stream, latest state)
→ Not suitable for: KStream (event stream, each change is an independent event)

References

  • Kafka: kafka.apache.org/documentation/#design
  • ISR: kafka.apache.org/documentation/#replication

Keywords: Kafka, partition, ISR, consumer group, rebalance, offset commit, log compaction

Related pages

© 2026 liz's doc