Message Queue Protocols

Messaging systems define not only how bytes are exchanged, but also who stores messages, when they are acknowledged, and how consumers recover. When comparing RabbitMQ, Kafka, and NATS, it is crucial to distinguish between standard protocols, products, and storage modes within those products; the "smart broker, dumb broker" or "minimal broker" abstractions are insufficient to explain real-world failure behaviors.

Within the same product, the impact of storage types, acknowledgment modes, and replication configurations often outweighs the differences between product names. RabbitMQ AMQP 1.0, NATS JetStream

First, Answer What "Success" Actually Acknowledges

When a producer hands bytes to a socket, it does not equal reliable receipt by the Broker; when the Broker returns success, it does not equal that subscribers have modified the business database; when a consumer acknowledges or commits an offset, it does not automatically prove that the local transaction has been committed. These boundaries occur across different processes and require explicit failure strategies.

If the consumer acknowledges first and then executes business logic, a crash may lead to missed processing; if the business is executed first and then acknowledged, a lost acknowledgment may lead to duplicates. Putting deduplication and local business effects within a single transaction can cover this local window, but external HTTP side effects still rely on external idempotency contracts or compensations. See Webhooks and Callbacks for specific receive models.

AMQP 0-9-1: Routing to Queues, Then Delivery from Queues

Publishers send messages to an exchange, and bindings determine which queues receive them. When a fanout exchange is bound to three queues, each queue can maintain its own pending consumption state; for three competing consumers on the same queue, they are typically sharing the delivery load of that queue, not each receiving a broadcast copy.

Queue durability, message persistence, replication, and publisher confirms are distinct configurations. A durable queue does not mean all its messages are naturally persisted in the same way; consumer ACKs do not indicate that replicas in other bound queues have been consumed. RabbitMQ Queues

Publish Confirms and Unrouted Messages

RabbitMQ publisher confirms are an extension of AMQP 0-9-1 used to confirm processing on the publish side. Messages that are not routed to any queue may still receive a publish confirm; discovering this situation requires the correct use of mechanisms like mandatory/return. Publish confirms and consumer ACKs are completely independent. RabbitMQ confirms

The scope of the consumption delivery_tag is the channel; it must be acknowledged on the same channel where it was received; old tags cannot be treated as persistent message IDs after reconnection. multiple=true acknowledges multiple unacknowledged deliveries up to the specified tag; if there is still unfinished work in between, it creates a window for missed processing.

In manual acknowledgment mode, connection or channel closure usually causes unacknowledged deliveries to be requeued; automatic acknowledgment mode does not provide the same protection. requeue can also alter the observed order; repeated nack/requeue can form hot loops, so attempt counts should be recorded, and different strategies should be adopted for transient failures versus unprocessable messages.

Prefetch is Not a Fairness Switch

prefetch limits the number of unacknowledged deliveries; 0 usually means no upper limit is set. RabbitMQ often applies non-global prefetch per consumer; if two consumers each have prefetch=10, do not mistake the total limit for the entire channel having only 10. Consumer Prefetch

Setting it to 1 limits in-flight messages per consumer, reducing the risk of slow consumers hogging a large batch of work, but throughput is still affected by acknowledgment round-trips and processing time, and it does not guarantee completely fair workload or time distribution among consumers. Setting it very high may cause client memory growth, re-injection storms, and tail latency.

Byte Frames and Business Messages Must Be Layered

AMQP 0-9-1 frames include type 1 B, channel 2 B, payload length 4 B, payload, and a trailing 0xce. A single publish may consist of method frames, content header frames, and multiple content body frames; it cannot be treated as a complete business message based on a single socket read or a single frame. Heartbeats are 8-byte frames of type 8, channel 0, with an empty payload. RabbitMQ Frame Format Errata

import struct
heartbeat = struct.pack('!BHI', 8, 0, 0) + b'\xce'
assert heartbeat.hex() == '08000000000000ce'
assert len(heartbeat) == 8
print('AMQP 0-9-1 empty heartbeat frame length check passed')

AMQP 1.0: Flow Control of Connections, Sessions, and Links

AMQP 1.0 is not just a version bump for 0-9-1. Endpoints establish sessions within a connection, then attach unidirectional links; messages are carried by transfers, with results expressed via delivery states and settlements, and link credit constrains sending permissions. It does not require all implementations to use identical exchange/queue management interfaces.

Two clients both called "AMQP" cannot necessarily communicate directly if their versions differ. During migration, one must check protocol versions, address mappings, authentication, acknowledgment modes, and product extensions; stating "both support AMQP" is insufficient to prove behavioral consistency. RabbitMQ AMQP 1.0 Concepts

Kafka: Log Positions, Group Progress, and Retention are Distinct States

Kafka appends records to partitions of a topic, with each record having an offset within that partition. In a regular consumer group, members are assigned partitions and pull records; different groups can independently read the same log. Group commit positions are typically maintained by the Kafka coordinator, not just stored within the consuming process.

Committing offsets does not delete the corresponding records. Logs are retained based on time, capacity, or compaction strategies; if a consumer falls behind until data is cleaned, the original position may no longer be replayable. The notion that "consumers track offsets themselves" does not imply that the Broker does not manage groups, transactions, or recovery states. Kafka Design

If an application fetches 100–103 for concurrent processing, and 100 and 102 complete first, the safe recovery position remains 101. Only when 101 is also completed can progress be made to 103; if 103 is committed first, the unfinished 101 will be skipped after a crash.

def resume_offset(first, completed):
    result = first
    while result in completed:
        result += 1
    return result

assert resume_offset(100, {100, 102}) == 101
assert resume_offset(100, {100, 101, 102}) == 103
assert resume_offset(100, {101, 102}) == 100
# Explicit assumption: only this contiguous segment of offsets; actual returned positions in compacted logs may have gaps.
print('Contiguous completed prefix and recovery position check passed')

This model uses contiguous integers to illustrate commit rules; real logs may have offset gaps due to transactions, compaction, or other reasons, and one should not wait indefinitely just because a specific integer is missing. The processed prefix should be maintained based on actual returned records and partition progress.

For non-contiguous offsets, batch boundaries, and handling of stale tasks after rebalancing, continue reading the distributed systems series Consumer Groups and Collaboration. This article retains minimal examples at the protocol level; full recovery progress management is elaborated there.

Kafka Wire Protocol is Not a Fixed Call Pipeline

Requests typically include a length prefix, API key/version, correlation ID, and version-specific headers; responses are correlated with requests via the correlation ID. ApiVersions is used for capability negotiation, Metadata provides node and partition information, Produce/Fetch handle records, and group protocols with OffsetCommit manage consumption coordination. Producers do not execute a Fetch before publishing each record.

Request/response schemas change with API versions; newer flexible versions also use compact encoding and tagged fields; a single fixed struct cannot parse all versions. Records may also be batched and compressed, so one request does not equal one message. Kafka Protocol

Acknowledgments for acks=all must be understood in conjunction with configurations like ISR and minimum in-sync replicas; it does not mean every configured replica is always in sync. Idempotent producers manage specific producer sessions and partition sequences; if the application arbitrarily recreates the producer and submits the same business event, business-level deduplication must still be considered. Kafka transactions can coordinate Kafka outputs and consumption positions, but do not automatically include external databases in the transaction.

102 is Done, Why Can We Only Commit 101?

If 100 and 102 complete first, the safe recovery position stops at 101; if 103 is mistakenly committed, 101 will be skipped after a crash. When 101 is completed, progress can be made to 103 in one step. Similar to TCP cumulative acknowledgments is the contiguous prefix constraint, but the difference is that "completion" here is defined by business consumption.

Core NATS: Text Commands and Byte-Delimited Payloads

Core NATS performs online pub/sub via subjects, and queue groups are used to distribute delivery within a group. It does not default to saving historical messages missed by offline subscribers; PING/PONG does not mean all subscribers have finished processing business logic.

Client protocol commands are represented in text, but payloads can be binary, and the length field is in bytes, not characters. SUB specifies the subject, an optional queue group, and a sid; PUB specifies the subject, an optional reply subject, and payload size. The sid is the subscription identifier for that connection, not a message ID. NATS Client Protocol

payload = 'Temperature=23'.encode('utf-8')
assert len(payload) == 9
packet = b'PUB sensors.temp ' + str(len(payload)).encode('ascii') + b'\r\n' + payload + b'\r\n'
line, rest = packet.split(b'\r\n', 1)
size = int(line.split()[-1])
assert rest[:size].decode('utf-8') == 'Temperature=23'
assert rest[size:] == b'\r\n'
print('NATS UTF-8 payload byte delimitation check passed')

NATS filters * match one token, and trailing > match one or more tokens; do not directly interchange them with MQTT #'s zero-level matching. request/reply uses a reply subject to return messages, but request timeouts can still occur after the remote end has already executed.

JetStream: Persistent Streams and Stateful Consumers

JetStream streams capture messages for specified subjects, with configurable memory/file storage, replication, and retention policies; consumers save delivery and acknowledgment states. It supports consumption methods like pull, and not all NATS functionality can be categorized as stateless, push-only, or guaranteed lowest latency. JetStream

Publishers confirm entry into the stream via JetStream publish acknowledgments; consumers advance consumer state via ACKs. Unacknowledged messages can be redelivered according to configuration; AckWait and maximum attempt counts need to be coordinated with processing time. Different retention policies have different impacts on whether messages are deleted after acknowledgment, and they cannot be uniformly interpreted as "ACK is just an offset" or "ACK always deletes."

Message deduplication windows and dual-confirmation features can reduce protocol-level duplicates, but they require corresponding configurations and APIs. Redelivery outside the window, application-created new events, or external side effects cannot be eliminated solely by claiming "exactly-once." JetStream Consumption Behavior

Comparing Using Backlog and Recovery Requirements

Assume an input of 1200 messages per second, with consumers stably processing 1000 per second, resulting in a backlog of 200 per second; after ten minutes, the backlog is 120,000 messages. If messages average 1 KiB including metadata, that is approximately 117.2 MiB, not counting replication replicas. Switching to a lighter wire protocol does not eliminate the long-term disparity between production and consumption.

Performance reports must list packet size, batching, compression, acknowledgments, replication, disk, and consumer workload. "Millions of messages per second, fixed 0.1 ms" without these conditions cannot serve as evidence for selection, and protocol complexity cannot be directly mapped to end-to-end latency.