18 min read #distributed-systems #Messages and Streams

On this page

Consumer Groups and Collaboration

A consumer group is a negotiation among consumers for partition ownership—an essential variant of distributed consensus. Eager vs. cooperative rebalancing determines whether the entire group stops or continues consuming during rebalancing.

Overview

Kafka Architecture discussed partitions and ISR—distributed mechanisms at the storage/replication layer. However, the consumer side also faces distributed coordination challenges: when multiple consumers share partitions of a topic, who consumes which partition? How are partitions reassigned if a consumer fails? Does consumption pause during reassignment? These questions are addressed by the consumer group—essentially, a "negotiation among a group of consumers for partition ownership," which is a true variant of distributed consensus.

How Consumer Groups Work

Kafka's consumption model: A partition can be consumed by only one consumer within a consumer group. This is the prerequisite for guaranteeing "message ordering within a partition." If two consumers read the same partition simultaneously, their respective offset ranges will be discontinuous, breaking the order.

Key point: The number of partitions determines the upper limit of parallelism. 6 partitions → maximum 6 consumers; if a 7th consumer joins, it will be idle (no partitions assigned to it). To increase consumption parallelism, you must add partitions, not consumers.

Rebalancing: Negotiating Ownership

When a consumer joins or leaves a group, a rebalance is triggered to reassign partition ownership. This is the most critical and error-prone part of the consumer group mechanism.

Eager Rebalance (Old Protocol): Stop and Reassign

1. New consumer C4 joins → Group coordinator notifies all consumers: release current assignments
2. All consumers stop consuming and release partition ownership
3. Reassignment: C1←P0,P1  C2←P2  C3←P3,P4  C4←P5
4. All consumers resume consumption from the last committed offset

This process is called stop-the-world: during steps 2-3, the entire consumer group stops consuming, leading to latency buildup. For latency-sensitive stream processing applications, every rebalance is an incident.

Cooperative Rebalance (New Protocol, KIP-429): Incremental

1. C4 joins → Coordinator calculates new assignment, notifying only consumers that "need to release partitions"
2. Affected consumers release the taken partitions but continue consuming unaffected partitions
3. New owners (C4) take over the released partitions
4. Throughout the process, unaffected consumers do not stop

The essential difference: eager makes everyone release and then reassign; cooperative only affects those impacted, leaving others running. It changes from "everyone stops" to "some keep running," significantly reducing the business impact of rebalancing.

Sticky Assignment: Maintain Existing Assignments Whenever Possible

Assignments after rebalancing consider not just "uniformity" but also "minimal change": if a consumer is already consuming a partition and does not need to release it, the assignment is kept unchanged. This is called sticky—avoiding unnecessary assignment changes and reducing consumer state switches.

Offset Management: The "Persistent Pointer" of Consumption Progress

Where the consumer has read needs to be recorded—this is the offset. There are two management methods:

	Auto Commit	Manual Commit
Timing	Automatically commits the current offset at time intervals (e.g., every 5 seconds)	Application explicitly calls `commitSync()` / `commitAsync()`
Risk	Process completes but commit interval hasn't arrived → crash → duplicate consumption (at-least-once)	Process completes → commit → crash → no duplicates; Process completes but crashes before commit → duplicates
Use Case	Tolerates small amounts of duplication	Requires fine-grained control (e.g., exactly-once consumption workflows)

Where offsets are stored: Kafka's internal __consumer_offsets topic (multi-partition, with replication). A consumer group's offset commit is a message written to this topic—Kafka uses its own storage mechanism to persist consumption progress.

Exactly-Once Consumption: Linking Consumer-Side Transactions

Message Semantics discussed producer-side exactly-once (idempotent producer + transactions). How is exactly-once achieved on the consumer side? The core is to place the commitment of consumption offsets and the output of processing results within the same transaction:

Consumer Transaction:
  1. Read message from topic-A (offset=N)
  2. Process → Write result to topic-B
  3. Commit offset (N+1) to __consumer_offsets
Steps 2 and 3 are within the same Kafka transaction:
    - Transaction success: Result written to topic-B, offset advances to N+1 → Consumption "occurred"
    - Transaction failure/Consumer crash: Result not written, offset rolls back to N → Message will be re-consumed

Key requirement: The consumer's isolation.level must be set to read_committed—only read messages from committed transactions, ignoring aborted transactions and incomplete dangling transactions.

This entire combination (idempotent producer + transactional producer + read_committed consumer + offset and output in the same transaction) constitutes Kafka's exactly-once semantics—not a theoretical "perfect once-only," but an engineering "effectively once-only," ensuring no duplicates or losses.

Comparison with RabbitMQ: Queue Model vs. Log Model

Another approach in the same chapter: RabbitMQ uses queues rather than logs. Messages are deleted after acknowledgment (not appended to an immutable log), with no replay/backtracking and no global ordering within partitions. Consumer group coordination also differs: RabbitMQ consumers do not negotiate partition ownership; instead, the broker distributes messages via round-robin or consistent-hash exchange—simpler, but lacking Kafka's "same key to same partition guarantees order."

The choice of Kafka's log model is because distributed stream processing requires replay and global ordering—two things the queue model cannot do. A Kafka partition is essentially a sharded, immutable, replayable log, and the consumer group is the "who reads which shard" negotiation layer on top of this log.

Common Issues and Fixes

Rebalance Storm: Consumers frequently join/leave (e.g., heartbeat timeout threshold too short, processing time too long → considered dead). Each rebalance causes the entire group to pause. → Set max.poll.interval.ms large enough (greater than the longest processing time), use cooperative rebalancing, and monitor rebalance frequency.
Partition Count < Consumer Count: Extra consumers remain idle → either add partitions or accept idle consumers (as hot backups).
Consumer Lag: Consumer processing speed cannot keep up with producer write speed → monitor lag (consumer offset vs. partition end offset), add consumers or partitions.
Auto Commit + Async Processing: Consumer submits offset after throwing message into thread pool → when async thread processing fails, the message is already "consumed" → loss. → Either process synchronously or manually commit after async processing completes.

References

Kafka Documentation: Consumer Group Protocol, KIP-429 (Incremental Cooperative Rebalancing)

Keywords: consumer group, rebalance, eager, cooperative (incremental), sticky assignment, stop-the-world, group coordinator, offset management, __consumer_offsets, exactly-once consumption, isolation.level, read_committed, transactional consumer, consumer lag, rebalance storm, Kafka vs RabbitMQ, log vs queue