--- title: Consumer Groups and Collaboration url: https://doc.liz6.com/en/distributed-systems/07-messages-and-streams/03-consumer-groups-and-collaboration locale: en area: distributed-systems tags: - distributed-systems - messages-and-streams date: 2026-06-30 modified: 2026-07-16 description: A consumer group is a negotiation among consumers for partition ownership—an essential variant of distributed consensus. Eager vs. cooperative rebalancing determines whether the entire group stops or continues consuming during rebalancing. --- # Consumer Groups and Collaboration > A consumer group is a negotiation among consumers for partition ownership—an essential variant of distributed consensus. Eager vs. cooperative rebalancing determines whether the entire group stops or continues consuming during rebalancing. ## Overview [Kafka Architecture](/distributed-systems/07-messages-and-streams/02-kafka-architecture.md) discussed partitions and ISR—distributed mechanisms at the storage/replication layer. However, the consumer side also faces distributed coordination challenges: when multiple consumers share partitions of a topic, who consumes which partition? How are partitions reassigned if a consumer fails? Does consumption pause during reassignment? These questions are addressed by the **consumer group**—essentially, a "negotiation among a group of consumers for partition ownership," which is a true variant of distributed consensus. ## How Consumer Groups Work Kafka's consumption model: **A partition can be consumed by only one consumer within a consumer group.** This is the prerequisite for guaranteeing "message ordering within a partition." If two consumers read the same partition simultaneously, their respective offset ranges will be discontinuous, breaking the order. Key point: The number of partitions determines the upper limit of parallelism. 6 partitions → maximum 6 consumers; if a 7th consumer joins, it will be **idle** (no partitions assigned to it). **To increase consumption parallelism, you must add partitions, not consumers.** ## Rebalancing: Negotiating Ownership When a consumer joins or leaves a group, a rebalance is triggered to reassign partition ownership. This is the most critical and error-prone part of the consumer group mechanism. ### Eager Rebalance (Old Protocol): Stop and Reassign ``` 1. New consumer C4 joins → Group coordinator notifies all consumers: release current assignments 2. All consumers stop consuming and release partition ownership 3. Reassignment: C1←P0,P1 C2←P2 C3←P3,P4 C4←P5 4. All consumers resume consumption from the last committed offset ``` This process is called **stop-the-world**: during steps 2-3, the **entire consumer group stops consuming**, leading to latency buildup. For latency-sensitive stream processing applications, every rebalance is an incident. ### Cooperative Rebalance (New Protocol, KIP-429): Incremental ``` 1. C4 joins → Coordinator calculates new assignment, notifying only consumers that "need to release partitions" 2. Affected consumers release the taken partitions but continue consuming unaffected partitions 3. New owners (C4) take over the released partitions 4. Throughout the process, unaffected consumers do not stop ``` The essential difference: eager makes everyone release and then reassign; cooperative only affects those impacted, leaving others running. **It changes from "everyone stops" to "some keep running,"** significantly reducing the business impact of rebalancing. ### Sticky Assignment: Maintain Existing Assignments Whenever Possible Assignments after rebalancing consider not just "uniformity" but also "minimal change": if a consumer is already consuming a partition and does not need to release it, the assignment is **kept unchanged**. This is called **sticky**—avoiding unnecessary assignment changes and reducing consumer state switches. ## Offset Management: The "Persistent Pointer" of Consumption Progress Where the consumer has read needs to be recorded—this is the offset. There are two management methods: | | Auto Commit | Manual Commit | |---|---|---| | Timing | Automatically commits the current offset at time intervals (e.g., every 5 seconds) | Application explicitly calls `commitSync()` / `commitAsync()` | | Risk | Process completes but commit interval hasn't arrived → crash → duplicate consumption (at-least-once) | Process completes → commit → crash → no duplicates; Process completes but crashes before commit → duplicates | | Use Case | Tolerates small amounts of duplication | Requires fine-grained control (e.g., exactly-once consumption workflows) | **Where offsets are stored:** Kafka's internal `__consumer_offsets` topic (multi-partition, with replication). A consumer group's offset commit is a message written to this topic—Kafka uses its own storage mechanism to persist consumption progress. ## Exactly-Once Consumption: Linking Consumer-Side Transactions [Message Semantics](/distributed-systems/07-messages-and-streams/01-message-semantics.md) discussed producer-side exactly-once (idempotent producer + transactions). How is exactly-once achieved on the consumer side? The core is to place the **commitment of consumption offsets** and the **output of processing results** within the same transaction: ``` Consumer Transaction: 1. Read message from topic-A (offset=N) 2. Process → Write result to topic-B 3. Commit offset (N+1) to __consumer_offsets Steps 2 and 3 are within the same Kafka transaction: - Transaction success: Result written to topic-B, offset advances to N+1 → Consumption "occurred" - Transaction failure/Consumer crash: Result not written, offset rolls back to N → Message will be re-consumed ``` Key requirement: The consumer's `isolation.level` must be set to `read_committed`—only read messages from committed transactions, ignoring aborted transactions and incomplete dangling transactions. This entire combination (idempotent producer + transactional producer + `read_committed` consumer + offset and output in the same transaction) constitutes Kafka's exactly-once semantics—not a theoretical "perfect once-only," but an engineering "effectively once-only," ensuring no duplicates or losses. ## Comparison with RabbitMQ: Queue Model vs. Log Model Another approach in the same chapter: RabbitMQ uses **queues** rather than logs. Messages are **deleted** after acknowledgment (not appended to an immutable log), with no replay/backtracking and no global ordering within partitions. Consumer group coordination also differs: RabbitMQ consumers do not negotiate partition ownership; instead, the broker distributes messages via round-robin or consistent-hash exchange—simpler, but lacking Kafka's "same key to same partition guarantees order." The choice of Kafka's log model is because distributed stream processing requires **replay** and **global ordering**—two things the queue model cannot do. A Kafka partition is essentially a sharded, immutable, replayable log, and the consumer group is the "who reads which shard" negotiation layer on top of this log. ## Common Issues and Fixes - **Rebalance Storm**: Consumers frequently join/leave (e.g., heartbeat timeout threshold too short, processing time too long → considered dead). Each rebalance causes the entire group to pause. → Set `max.poll.interval.ms` large enough (greater than the longest processing time), use cooperative rebalancing, and monitor rebalance frequency. - **Partition Count < Consumer Count**: Extra consumers remain idle → either add partitions or accept idle consumers (as hot backups). - **Consumer Lag**: Consumer processing speed cannot keep up with producer write speed → monitor lag (consumer offset vs. partition end offset), add consumers or partitions. - **Auto Commit + Async Processing**: Consumer submits offset after throwing message into thread pool → when async thread processing fails, the message is already "consumed" → loss. → Either process synchronously or manually commit after async processing completes. ## References - **Kafka Documentation**: Consumer Group Protocol, KIP-429 (Incremental Cooperative Rebalancing) *Keywords: consumer group, rebalance, eager, cooperative (incremental), sticky assignment, stop-the-world, group coordinator, offset management, __consumer_offsets, exactly-once consumption, isolation.level, read_committed, transactional consumer, consumer lag, rebalance storm, Kafka vs RabbitMQ, log vs queue*