Distributed Transactions

Suppose placing an order requires creating an order, reserving inventory, and calling a payment service simultaneously. If one step fails or the response is lost, the system may only complete part of the operation. Distributed transactions and workflow design must answer: which effects must be committed together, which intermediate states are permissible, and how to reliably advance incomplete operations to an explainable result.

Distributed systems can provide ACID transactions, but this requires appropriate protocols and support from participating resources. It is incorrect to say that cross-machine "global ACID" is naturally absent, nor should one automatically include arbitrary external HTTP calls in transactions just because the database supports cross-shard transactions.

First, Distinguish Atomicity, Isolation, and Business Recovery

Atomic commit addresses whether multiple participants make a consistent decision to commit or abort the same transaction. Isolation determines what concurrent transactions can see and whether they are equivalent to some serial execution. Durability concerns how successful results are persisted across failures. These require different mechanisms working in concert.

For example, if two shards both commit a transfer, satisfying the common commit requirement does not independently prove that concurrent balance checks are serializable. 2PC is an atomic commit protocol; it does not automatically provide all isolation levels merely through two rounds of messaging. Locks, MVCC, timestamps, or conflict validation are still needed to handle concurrency control.

2PC: Why a "Prepared" Phase is Necessary

The coordinator cannot simply send ordinary COMMIT messages sequentially: after the first resource succeeds, the second might fail to commit due to constraint conflicts. The Prepare phase requires each participant to first confirm and persistently retain the ability to "commit according to the final decision."

Before participants vote YES, they must complete the persistence and constraint checks required by the protocol, and retain the necessary resources. Lock-based implementations may continue to hold relevant locks; this does not mean the entire database is locked. The coordinator must reliably persist the commit decision before issuing commit notifications; otherwise, it may be unable to explain already-committed participants upon recovery.

Where does the blocking come from?

Consider A and B both entering the Prepared state. The coordinator has decided on COMMIT. A receives the notification and commits, but the notification sent to B is lost. After B times out, it cannot rollback on its own, as this would conflict with A's result. Even if B receives no commit message, it cannot infer that no commit decision exists.

If a participant has not yet made a YES commitment, the protocol may allow it to reject or abort; if it can obtain a reliable decision from the recovery log or other participants, it can proceed. The blocking issue in classic 2PC is: some prepared participants must wait when they cannot obtain a decision. It is not the case that "if the coordinator fails, all participants in any state wait forever."

After recovery, the coordinator can re-send the persisted decision, and participants can be recovered by the transaction manager. PostgreSQL's prepared transactions are a mechanism for external transaction managers; long-standing prepared transactions retain resources, so reliable cleanup and recovery management are required. PREPARE TRANSACTION documentation

Replicating coordinator state can reduce wait times caused by single-machine failures, but it cannot magically recover necessary resources that are unreachable. The relationship between consensus and transaction commit is discussed in Gray & Lamport's paper. Administrator manual decisions to commit or rollback belong to exception handling and should not be the default step for normal recovery.

The same silence seen by a Prepared participant can correspond to two different histories

From B's perspective, two executions look identical: it has persisted its preparation, voted YES, and then waits for a final notification. However, other parts of the system may be in different states:

If B autonomously ABORTs based on timeout, it breaks the atomicity of Execution 1; if it autonomously COMMITs, it violates the commit conditions in Execution 2. Relying solely on B's local silence does not allow choosing a terminal state that is safe for both histories.

Therefore, recovery must seek a reliable decision or a protocol-allowed basis for decision, rather than guessing the answer after waiting for a certain duration. Committed participants can provide reliable evidence of commit to aid recovery; if all reachable participants only know about the Prepared state, the decision may still be missing. Retaining resources in the prepared state is the cost of this uncertainty, not a window that can be eliminated by ordinary RPC retries.

The same silence may hide two different decisions

The waiting here is not an animation missing the next step, but rather B genuinely lacks sufficient knowledge. Setting the remote side to "COMMIT persisted" and "ABORT persisted after another participant rejected" are two different executions; B still only has the same Prepared evidence before the notification arrives. Switching remote histories is a comparison of two executions, not an allowance for the same transaction to modify a decision already made.

Saga: Implementing Business Processes with a Series of Local Transactions

Saga breaks a process into independently commitable steps. Taking the example order process: create a pending payment order, request deduction, arrange shipment. If the deduction explicitly fails, the order can be changed to cancelled; if the deduction succeeds but shipment cannot ultimately be fulfilled, a refund can be requested and fulfillment marked as failed.

Compensation is a new business operation, not restoring the database to its past byte state. A refund can offset the main business effect of the deduction, but payment records, notifications, and audit history still exist. Concurrent business processes may have already read the intermediate state, so you cannot "undo" by overwriting the entire object with an old snapshot. This semantic distinction of compensation is emphasized in the original Sagas paper.

Intermediate States Must Be Part of the Business Model

Orders can have states like "pending payment," "payment result pending confirmation," "refund in progress," etc., with restrictions on which states allow shipment. Simply calling reverse interfaces upon failure cannot prevent other processes from continuing execution based on intermediate results before compensation occurs.

Compensation can also fail, timeout, or wait for a long time. It is necessary to persistently record steps, request IDs, results, and compensation progress, retry according to policies, and enter manual processing states when necessary. Network timeouts first mean the result is unknown; one should query or retry for confirmation using stable operation IDs, rather than immediately treating it as "never deducted" and creating a new payment.

Both require reliable state and message handling. Event-driven design does not automatically eliminate coordination logic; it merely distributes it across services.

TCC: Secure Resource Reservation First

Try-Confirm-Cancel defines three actions for business resources: Try checks and reserves, Confirm consumes the reservation, Cancel releases the reservation. For example, with 10 items in stock, after Try reserves 2, the available stock becomes 8; Try has already modified the resource state, it is not "nothing happened."

To make Confirm fulfillable, Try must convert business preconditions into reliable reservations, preventing other requests from consuming the same resource. Confirm may still encounter temporary technical failures, so retries are needed; if resources expire unconditionally before confirmation, expiration and the final decision must be coordinated, otherwise the promise that "Try success implies Confirmability" does not hold.

Three common boundaries can be explained by the same persistent state machine:

• Idempotency: Duplicate arrival of Try, Confirm, or Cancel for the same branch does not repeatedly occupy, consume, or release resources.
• Empty Rollback: If Cancel arrives before Try, it is still recorded as cancelled, and no resource release is needed.
• Anti-Hanging: Late Trys after cancellation must be rejected; they cannot re-establish a reservation with no one to clean it up.

Confirm and Cancel must adhere to a consistent final decision and avoid both taking effect through atomic state transitions. If business needs to be reversed after confirmation, a new refund or return process should be entered; TCC cannot simply Cancel at any moment.

Check TCC Branches Using "Available + Reserved + Consumed = Initial Stock"

Below, a branch call is treated as an atomic local state transition, with an initial stock of 10. Branch identity is bound to quantity, and all actions check parameter consistency; Confirm returns failure if there is no successful Try, and Cancel can record an empty cancellation first. Returning True indicates the semantic requirement of the action is met; repeated calls do not mean the resource is modified again.

class InventoryTCC:
    def __init__(self, stock=10):
        self.initial = self.available = stock
        self.consumed = 0
        self.branches = {}  # branch_id -> (state, quantity)

    def call(self, action, branch_id, quantity):
        if action not in {"try", "confirm", "cancel"}:
            raise ValueError("Unknown action")
        if type(quantity) is not int or quantity <= 0:
            raise ValueError("Quantity must be a positive integer")
        previous = self.branches.get(branch_id)
        if previous is not None and previous[1] != quantity:
            raise ValueError("Quantity cannot be changed for the same branch")
        state = previous[0] if previous else "new"
        if action == "try":
            if state == "cancelled":
                return False
            if state in {"reserved", "confirmed"}:
                return True
            if self.available < quantity:
                return False
            self.available -= quantity
            self.branches[branch_id] = ("reserved", quantity)
        elif action == "confirm":
            if state == "confirmed":
                return True
            if state != "reserved":
                return False
            self.consumed += quantity
            self.branches[branch_id] = ("confirmed", quantity)
        else:
            if state == "confirmed":
                return False
            if state == "cancelled":
                return True
            if state == "reserved":
                self.available += quantity
            self.branches[branch_id] = ("cancelled", quantity)
        return True

    def check(self):
        reserved = sum(q for state, q in self.branches.values() if state == "reserved")
        assert self.available >= 0 and self.consumed >= 0
        assert self.available + reserved + self.consumed == self.initial
        return self.available, reserved, self.consumed

stock = InventoryTCC()
assert stock.call("try", "order-1", 2)
assert stock.call("try", "order-1", 2)
assert stock.check() == (8, 2, 0)
assert stock.call("confirm", "order-1", 2)
assert stock.call("confirm", "order-1", 2)
assert not stock.call("cancel", "order-1", 2)
assert stock.check() == (8, 0, 2)

assert stock.call("cancel", "order-2", 3)  # Empty rollback, leaves terminal state
assert not stock.call("try", "order-2", 3)  # Reject late Try
assert stock.check() == (8, 0, 2)
assert stock.call("try", "order-3", 4)
assert stock.call("cancel", "order-3", 4)
assert stock.call("cancel", "order-3", 4)
assert stock.check() == (8, 0, 2)
assert not stock.call("try", "too-large", 9)
assert stock.check() == (8, 0, 2)

Try moves quantity from available to reserved; Confirm moves from reserved to consumed; Cancel moves from reserved back to available. If repeated Cancel increases available again, it creates inventory out of thin air, and the conservation equation catches this error immediately. Similarly, empty cancellation cannot increase inventory because there was no reservation to begin with.

This is a single-threaded in-memory model, without disk persistence, process crash handling, or concurrency locks. Real implementations must atomically commit branch state, quantity, and inventory modifications together, and verify that the operation indeed comes from a coordinator authorized to advance that transaction. When multiple transactions share inventory, available quantity must be checked atomically; one cannot read inventory and then unconditionally deduct.

More importantly: local terminal state mutual exclusion does not equal global decision consistency. If the coordinator erroneously sends Confirm to A and Cancel to B, both can legally enter their respective terminal states, and the system is still not atomic; terminal state checks only prevent the same branch from changing its mind repeatedly. The coordinator's persistent decision, recovery, and retry are another layer of necessary mechanisms; Seata's TCC mode also incorporates custom branch behavior into global transaction management. TCC Branches and Global Transactions

Records of cancelled branches cannot be deleted immediately, or late Trys will be treated as new requests. The retention period must cover the allowed retry and replay window, or the protocol must prove that old branches can never appear again; this is the same retention issue as message consumption deduplication.

Will retrying once create inventory out of thin air?

The conservation equation turns retry errors into checkable quantity changes: after any valid action, the sum of available, reserved, and consumed should remain 10. Cancel then Try can check for anti-hanging; repeated Cancel can check for erroneous returns; Cancel after Confirm must be rejected. Satisfying these rules locally does not replace the need for consistent decisions from a global coordinator.

$$A+R+C=10$$

Idempotency: One Business Request Produces One Effect

Idempotent interfaces need to identify "this is the same request," not just make SQL look repeatable. Use (business scope, idempotency_key) to uniquely identify operations, reliably saving request parameter summaries, business effects, and results together.

Deduplication records and business modifications need the same atomic boundary. Marking as processed first then executing may miss execution on crash; executing first then marking may result in duplicate execution on crash. For external services, use stable operation identifiers and result query mechanisms supported by the other party. Stripe's API demonstrates specific contracts such as saving results by idempotency key, validating parameters, and retention periods. Stripe Idempotent Requests Documentation

Optimistic locking WHERE version = expected is used to detect contention, but does not equal complete deduplication: an update affecting 0 rows may be because the original request succeeded, or because another request modified the record. The client still needs to identify its own operation result. Using the same idempotency key for different request parameters should also error, rather than silently returning unrelated results.

Outbox: Commit Business State and Pending Events Together

If the order database and message system cannot commit to a common transaction, you can modify the order and insert an Outbox event simultaneously in one local transaction, and then have a forwarder send the message. This way, successful local business changes have a persistent record of pending sends.

If the forwarder crashes after the message is sent but before the send status is updated, it will resend. Therefore, Outbox usually requires idempotent processing by consumers, and does not naturally provide "exactly once" for external side effects. See Message Semantics for a complete example.

Quick Reference

Selection should start from constraints: when strong coupling data can be placed within the same transaction boundary, prioritize evaluating this simplification; when crossing resources is necessary, then clarify participant capabilities, allowed intermediate states, failure recovery, and business compensation, rather than simply choosing solutions based on "strong consistency" or "high availability" labels.

References

• Consensus on Transaction Commit.
• Original Sagas Paper.
• PostgreSQL PREPARE TRANSACTION.
• Stripe Idempotent Requests.

Keywords: distributed transaction, atomic commit, 2PC, Saga, TCC, compensation, idempotency, outbox