Transactional Outbox

Overview

The transactional outbox pattern records a business change and a message to be delivered later within the same database transaction. A separate relay reads committed messages and sends them to a broker or receiving service.

Leaving one operation's effects in two systems creates a failure window. If a process stops after committing an order but before sending its message, the order exists while other services remain unaware of it. Sending first creates the opposite risk: another service can act on an order that never commits.

An outbox keeps a durable record of the work to send. Committing the order and event together leaves the delivery task in the database even if the relay stops temporarily. The business request's execution lifetime and the message's delivery lifetime separate at this boundary.

Understanding the pattern starts with its transaction boundary. Which records commit together, who delivers them, and how receivers handle duplicates matter more than a queue-table name or scheduler syntax.

Timeline

• 1981: Jim Gray's VLDB paper The Transaction Concept: Virtues and Limitations discussed the boundary between transaction-protected data and external actions.¹

• 2003: Gregor Hohpe's integration presentation compared file transfer, shared databases, remote invocation, and messaging. This date comes from the slides' copyright notice and is distinct from the book's publication date.²

• October 2018: Manning published Richardson's Microservices Patterns, a book account of service-database updates and event publication.³

Protected data and external actions (1981)

Jim Gray's 1981 paper divides actions into three categories. Unprotected actions need not be undone or redone after an abort or during recovery; protected actions can and must be undone or redone; real actions cannot be undone once performed. Real device actions, such as dispensing cash, are deferred until commit and named, for example with sequence numbers, so duplicate executions can be discarded.¹

Message operations within the same transaction system are protected actions in Gray's classification, alongside database operations. Messages sent to a broker that does not participate in the business database's transaction fall outside that local recovery scope. A process stopping between an order's commit and message transmission exposes the gap. Calling the operations in sequence cannot commit them together. Distributed transactions coordinate participating resources; an outbox divides the problem around a locally recorded intent to send.¹⁴

Enterprise integration and transactional messaging (2003)

Gregor Hohpe's 2003 presentation records the need to consider delivery timing and coupling when connecting applications. Hohpe and Bobby Woolf's Transactional Client pattern makes the client's messaging session transactional so that the client can define its boundaries. A sent message enters the channel on commit, while a received message is removed on commit.²⁵

The public pattern explanation concerns the session's messaging operations. Including changes in a separate business database requires examining participation by that resource and its transaction manager. A messaging transaction and a service's database transaction can have different boundaries; an outbox connects a committed business change to a record that can be delivered later.

Service databases and event publication as a pattern (2018)

Chris Richardson's Microservices Patterns, published in October 2018, discusses the problem of service-database updates and event publication. His current catalog describes recording business data and a pending event in one local transaction as Transactional Outbox and also lists the name Application events. A separate relay reads committed events and sends them to a broker. The intent survives the request that created it, allowing delivery to continue later.⁴³

The records committed together belong to the producer's database, which sets the scope of that commit. The receiver commits its own state separately, with the broker carrying messages between them.

Table polling and log-based delivery

Richardson's catalog distinguishes polling from transaction-log tailing as relay implementations. Polling queries committed outbox rows; log tailing reads database change records to discover pending delivery.⁴

Debezium's Outbox Event Router routes outbox records obtained through change data capture. In its default configuration, the event ID (id) becomes a message header, the aggregate identifier (aggregateid) becomes the message key, and the aggregate type (aggregatetype) contributes to the topic name. The payload becomes the message value. Other columns, such as event type, can be added to headers or an envelope through table.fields.additional.placement.⁶

Polling depends on query intervals, indexes, and work claiming. Log-based delivery depends on retained logs, checkpoints, and connector lifetimes. Both need a record and a position from which delivery can resume after interruption.

Duplicate delivery and idempotent receivers

A relay can transmit a message and stop before recording that it sent it. The same event can then be transmitted again. AWS's pattern guidance discusses duplicate delivery and receiver idempotency.⁷

A receiver can use an event ID to recognize an event it has processed. Recording that identity and the business effect in one local transaction can prevent redelivery from creating the effect again. The identity's scope and retention period belong to the same duplicate-detection contract.

This protection covers effects recorded together in that database. Payments or email in another system also need that system's request-identity and outcome contract.

Ordering, retention, and related patterns

Ordering matters when events such as creating and cancelling one order have different effects in different sequences. Parallel consumers need a defined aggregate scope within which order is preserved. Message keys and partitions are tools for implementing it.⁴⁶

When event records are deleted affects relay progress and recovery. Deleting a committed event before delivery removes the record needed to continue; retaining every delivered event increases storage costs. Deletion conditions follow acknowledgement positions and recovery requirements.

An outbox handles delivery of state changes, while a saga coordinates business steps across services. In event sourcing, the events themselves are the source of business state. The distinction depends on which records define that state, rather than on the presence of a message table.

An example of two storage boundaries

The diagram groups the producer's and receiver's local transactions separately. It shows a relay calling the receiver directly without a broker, so sent is marked after the receiver processes the event. With a broker, the relay marks sent after the broker acknowledges receipt. The PostgreSQL code below executes only the producer's order and event records; the relay and receiver are outside this code example.

CREATE TEMP TABLE orders (id integer PRIMARY KEY, item text NOT NULL);
CREATE TEMP TABLE outbox (
  event_id text PRIMARY KEY,
  order_id integer NOT NULL,
  event_type text NOT NULL
);

BEGIN;
INSERT INTO orders VALUES (1, 'book');
INSERT INTO outbox VALUES ('event-1', 1, 'OrderPlaced');
COMMIT;

SELECT id, item FROM orders;
SELECT event_id, order_id, event_type FROM outbox;

The SELECT statements return one order and the event-1 event. BEGIN and COMMIT establish those two records together. The diagram's handle_event and receiver database represent a later, separate processing boundary. The TEMP tables are for practice and are visible only in the creating session; a real relay running in another session needs ordinary tables it can read.