Message Semantics

Message semantics describe what a system can still guarantee after retries, disconnections, or process crashes. Before discussing "exactly-once," we must clarify what is being counted: the number of network sends, the number of consumer executions, or the number of business operations that ultimately take effect in the database. These three can differ.

This article first explains the problem using a scenario of a lost acknowledgment, then discusses the three semantics and their implementation mechanisms. For Kafka's log structure, see Kafka Architecture; for partition takeover and offset management, see Consumer Groups and Collaboration.

Where the Problem Comes From: Not Receiving an Acknowledgment Does Not Mean Failure

Suppose a consumer processes event E42, deducts one item from inventory, and then acknowledges to the message system:

The message system only sees "no ACK received." The consumer might not have processed it at all, or it might have finished processing, but the acknowledgment was lost. The former requires retry; the latter means a retry could deduct inventory a second time. Simply extending the timeout cannot eliminate this uncertainty window.

The same problem exists in the producer-to-broker link. Producer acknowledgment indicates that the message has met certain acceptance conditions on the broker side; consumer acknowledgment indicates that processing progress can be advanced. The two types of acknowledgments cannot replace each other. RabbitMQ explicitly distinguishes between Publisher Confirms and Consumer Acknowledgements.

Accurate Meanings of the Three Semantics

"At-least-once" relies on message retention, persistence, recovery capabilities, and retry strategies. Queue expiration with deletion, exhaustion of retry counts, or loss of all replicas are not unconditionally guaranteed. "At-most-once" is not simply achieved by turning off retries in one place: if the application layer resubmits the same business request, it may still generate two different messages.

Exactly-once does not require the processing function to physically run only once. The first execution might roll back, and the second execution commits; the outside world only sees one result. You cannot directly derive "exactly-once effect is impossible" from FLP: FLP discusses the termination of deterministic consensus under specific asynchronous fault models and cannot replace the analysis of transaction boundaries and visible results here.

How Acknowledgment Order Determines the Failure Window

Let business effect be E, and acknowledgment or progress commit be C. When they are independent, there are only two orders:

Therefore, simply swapping the order cannot eliminate both omissions and duplicates. The system must further ensure that E and C happen atomically, or allow E to be repeated but convert the repetition into an operation with no additional side effects. This is the role of transactions and idempotency.

Idempotency: How to Get the Same Business Result from Repeated Requests

Mathematically, an operation f is idempotent if f(f(x)) = f(x). Setting the state to CONFIRMED is idempotent if we only consider that field assignment; adding one to a quantity is not. However, if sending emails, charging, or writing audit events happens alongside the assignment, the entire processing function may not be idempotent.

Business deduplication requires stable event IDs. Retries of the same event must reuse the ID; generating a new UUID for each retry will invalidate deduplication. A common key is (consumer_name, event_id): the same event can be processed once by each different business consumer.

Below is an example of atomic consumption in PostgreSQL. The premise is that product_id is unique in the inventory table, and processing success is defined as an increase in inventory; message format validation is completed before the transaction.

CREATE TABLE inventory (
    product_id bigint PRIMARY KEY,
    count bigint NOT NULL
);

CREATE TABLE processed_events (
    consumer_name text NOT NULL,
    event_id text NOT NULL,
    PRIMARY KEY (consumer_name, event_id)
);

-- Parameter example: consumer_name='inventory', event_id='E42'.
BEGIN;
WITH first_delivery AS (
    INSERT INTO processed_events (consumer_name, event_id)
    VALUES ('inventory', 'E42')
    ON CONFLICT DO NOTHING
    RETURNING event_id
)
INSERT INTO inventory (product_id, count)
SELECT 42, 1 FROM first_delivery
ON CONFLICT (product_id)
DO UPDATE SET count = inventory.count + EXCLUDED.count;
COMMIT;
-- Acknowledge the message only after successful COMMIT.

If the first insertion of the deduplication key is successful, inventory increases; on the second attempt, first_delivery is empty, and inventory is no longer modified. If the inventory update fails, the entire transaction rolls back, and the deduplication key is not left behind. This relies on PostgreSQL's ON CONFLICT and RETURNING semantics.

This example also reveals two boundaries: if the deduplication record and inventory are in two independent databases, the above atomicity does not exist; if the deduplication record for E42 is cleared in advance, replaying the old message might take effect again. The deduplication retention period must cover the replay range allowed by the business.

Versioned conditional updates can also prevent overwrites, but a version mismatch only indicates that the state has changed, which is insufficient to prove "this event has been processed." Event identity and concurrent version resolution address two different problems.

Hands-on Observation: What State is Left Behind by Failures Before and After Transaction Commit

Simulate an inventory consumer using a local SQLite file, with an initial inventory of 100, and E42 deducting 3. Here, the event content is simplified to an integer delta; the deduplication record also saves the delta to prevent silently swallowing different operations that might accidentally correspond to the same ID. Real events should also compare products, tenants, and business parameters, or compare normalized content hashes.

import sqlite3
from pathlib import Path
from tempfile import TemporaryDirectory

class Interrupted(Exception):
    pass

def consume(path, event_id, delta, fail_at=None):
    db = sqlite3.connect(path, isolation_level=None)
    try:
        db.execute("BEGIN IMMEDIATE")
        inserted = db.execute(
            "INSERT INTO done VALUES (?, ?) ON CONFLICT(id) DO NOTHING",
            (event_id, delta),
        ).rowcount
        if not inserted:
            old_delta = db.execute(
                "SELECT delta FROM done WHERE id = ?", (event_id,)
            ).fetchone()[0]
            if old_delta != delta:
                raise ValueError("Same event ID corresponds to different content")
        else:
            if fail_at == "after_claim":
                raise Interrupted()
            changed = db.execute(
                "UPDATE stock SET n = n + ? WHERE n + ? >= 0",
                (delta, delta),
            ).rowcount
            if changed != 1:
                raise ValueError("Insufficient inventory")
            if fail_at == "after_effect":
                raise Interrupted()
        db.execute("COMMIT")
        if fail_at == "after_commit":
            raise Interrupted()  # Result committed, but caller didn't get success response
    except Exception:
        if db.in_transaction:
            db.execute("ROLLBACK")
        raise
    finally:
        db.close()

def state(path):
    with sqlite3.connect(path) as db:
        return (db.execute("SELECT n FROM stock").fetchone()[0],
                db.execute("SELECT count(*) FROM done").fetchone()[0])

with TemporaryDirectory() as directory:
    for cut in ("after_claim", "after_effect", "after_commit"):
        path = Path(directory) / f"{cut}.sqlite"
        with sqlite3.connect(path) as db:
            db.executescript("""
                CREATE TABLE stock (n INTEGER NOT NULL CHECK(n >= 0));
                INSERT INTO stock VALUES (100);
                CREATE TABLE done (id TEXT PRIMARY KEY, delta INTEGER NOT NULL);
            """)
        try:
            consume(path, "E42", -3, cut)
        except Interrupted:
            pass
        assert state(path) == ((97, 1) if cut == "after_commit" else (100, 0))
        consume(path, "E42", -3)  # Caller doesn't know the previous result, so retries
        consume(path, "E42", -3)  # Message delivered again
        assert state(path) == (97, 1)
        for event_id, delta in (("E42", -4), ("E43", -200)):
            try:
                consume(path, event_id, delta)
            except ValueError:
                pass
            else:
                raise AssertionError("Should reject content conflict or insufficient inventory")
        assert state(path) == (97, 1)

Each call reopens the database, observing the state readable after commit. The experiment injects Python exceptions and explicitly rolls back, demonstrating transaction boundaries; it does not simulate power loss, disk failure, or message broker issues. SQLite allows only one write transaction, and BEGIN IMMEDIATE may return busy due to other writers; production consumers also need to handle such retryable failures. See SQLite Transaction Documentation for specific transaction behaviors.

The third row is the uncertainty window truly resolved by idempotent records. If the dedup key is committed in an independent transaction first, the first row becomes "processed but inventory not changed"; if the business logic commits independently first, the second row becomes "inventory changed but no dedup evidence." Putting two tables in the same database is not enough; they must be committed by the same transaction.

Insufficient inventory is defined here as a rejection of execution, so no successful dedup record is left; the business system can record this as a queryable rejection result, but needs to clarify whether this result allows retries later. If the requirement is for retries to return the first generated order number, receipt, or rejection reason, that result must also be saved in the transaction; recording only an "appeared" ID is insufficient to reconstruct the response.

After Response Loss, Will Redelivery Deduct Again?

Counting the "number of processing attempts" and "number of persistent deduction times" for the same E42 allows for an accurate discussion of exactly-once effects. A crash can occur before commit, or after commit but before ACK: the former discards the stash, while the latter preserves the database result but leaves the broker able to redeliver. Switching strategies resets the entire history; compare whether the dedup record shares an atomic boundary with the effect.

Kafka Idempotent Production and Transactions

Kafka's idempotent production brings producer identity, epoch, and partition-internal sequence numbers into the protocol to identify duplicate batches generated by client send retries. This is not equivalent to deduplication by business content: if the application sends the same order as a new record, it may still be appended normally. See Producer Configs for related constraints.

Transactions further bring the output of multiple partitions and consumption progress into a single commit. A Kafka-to-Kafka transformation can be written as:

On crash, the transaction may not have been committed, or it may have been committed but the response lost; recovery is handled by the transaction protocol. Downstream systems use read_committed to exclude records from uncommitted or aborted transactions. "Result visibility" and "where to resume on recovery" should be viewed as the same atomic decision. See Kafka Message Delivery Semantics / Transactions for mechanisms and limitations.

Kafka transactions do not automatically include external SQL updates, HTTP requests, or email sends. The database side must use its own transactions and deduplication; external interfaces need stable idempotent keys or queryable business state to resolve uncertain results.

Outbox: How to Reliably Emit Events After Database Commit

"Write to database first, then send message" also has two independent commit points: if the process crashes after database success, the message may never be sent. Conversely, sending the message first might publish a business fact that is never committed.

The Outbox pattern writes business changes and pending events into the same database transaction, which is read by an independent forwarder for delivery:

If the forwarder crashes between "sent" and "marked," it will resend. Thus, Outbox mainly solves miss-sends, while consumer deduplication handles duplicates. Only when the two work together is the complete link formed. See Distributed Transactions for more general transaction trade-offs.

Quick Judgment of Guarantee Boundaries

When checking a message link, determine in sequence: who generates stable event IDs; which step might retry; what the ACK actually confirms; whether business effects and progress can share a transaction; how to deduplicate external side effects; and how long deduplication records and original messages are retained.

If these questions do not have clear answers, simply writing "enable exactly-once" does not indicate what guarantees the system actually provides.

References

• RabbitMQ: Acknowledgment Mechanism
• Kafka: Design and Message Semantics
• Kafka: Producer Configuration
• SQLite: Transactions and Explicit Rollback
• PostgreSQL: INSERT

Keywords: Message Semantics, Acknowledgment, Retry, Idempotency, Deduplication, exactly-once, Transaction Boundary, Outbox