The transactional outbox: stop losing events between your database and Kafka

This is one of the most common bugs in event-driven systems, and it hides in two innocent lines:

orderRepository.save(order);                 // 1. commit to the database
kafkaTemplate.send("orders", orderCreated);  // 2. publish the event

If the process dies between line 1 and line 2 — a deploy, an OOM kill, a lost connection to the broker — the order exists, but the event never goes out. Billing never bills, the warehouse never ships, and nothing errors, because from every service’s point of view nothing happened.

Swapping the lines doesn’t help. Publish first, and if the database commit then fails you’ve announced an order that doesn’t exist.

The underlying problem is a dual write: two systems, no shared transaction. You can’t make a Postgres commit and a Kafka publish atomic without distributed transactions, which most teams rightly avoid. But you can make two writes to the same database atomic. That’s the whole idea of the outbox.

The pattern

1. In the same transaction as the business change, insert a row into an outbox table describing the event.
2. A separate relay reads new outbox rows and publishes them to Kafka.
3. Once Kafka acknowledges, the relay marks the row as published (or deletes it).

If the service crashes after the commit, the outbox row is still there and the relay publishes it later. If the transaction rolls back, the outbox row rolls back with it. The event exists if and only if the order does.

The table and the write

CREATE TABLE outbox (
    id            BIGSERIAL    PRIMARY KEY,
    aggregate_id  TEXT         NOT NULL,      -- e.g. the order id; used as the Kafka key
    event_type    TEXT         NOT NULL,      -- OrderCreated, OrderCancelled, ...
    payload       JSONB        NOT NULL,
    created_at    TIMESTAMPTZ  NOT NULL DEFAULT now(),
    published_at  TIMESTAMPTZ
);
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;

@Transactional
public Order placeOrder(PlaceOrder cmd) {
    Order order = orders.save(Order.from(cmd));
    outbox.save(new OutboxEvent(order.getId(), "OrderCreated", toJson(order)));
    return order;   // both rows commit together, or neither does
}

The partial index keeps the relay’s query fast no matter how many published rows you keep around.

Relay option 1: polling

The simplest relay is a scheduled loop:

@Scheduled(fixedDelay = 500)
@Transactional
public void relay() {
    List<OutboxEvent> batch = jdbc.query("""
        SELECT id, aggregate_id, event_type, payload
        FROM outbox
        WHERE published_at IS NULL
        ORDER BY id
        LIMIT 100
        FOR UPDATE SKIP LOCKED
        """, outboxRowMapper);

    for (OutboxEvent e : batch) {
        kafka.send("orders", e.aggregateId(), e.payload()).get();   // wait for the ack
    }
    jdbc.update("UPDATE outbox SET published_at = now() WHERE id = ANY(?)", ids(batch));
}

FOR UPDATE SKIP LOCKED lets you run several relay instances: each one locks a batch and the others skip it instead of waiting or double-publishing. Using aggregate_id as the Kafka key keeps all events for one order on one partition, so consumers see them in order.

Polling is easy to understand, needs no new infrastructure, and its latency (half a second here) is fine for most business events.

Relay option 2: change data capture

If you need lower latency or you don’t want polling load on the database, use change data capture: a tool like Debezium reads the database’s replication log (the Postgres WAL, the MySQL binlog) and publishes each new outbox row. Debezium even ships an “outbox event router” that maps outbox columns onto Kafka topics and keys.

The trade-off is operational: you now run Kafka Connect and Debezium, and you need to manage replication slots so they don’t hold back WAL cleanup. For a team already running Kafka Connect it’s the better relay. For a team that isn’t, polling is usually enough to start.

The part people forget: duplicates

The outbox guarantees at-least-once, not exactly-once. If the relay publishes a batch and crashes before marking it published, the next run publishes it again. So consumers must be idempotent:

• Give every event a unique ID (the outbox id, or a UUID in the payload).
• In the consumer, record processed event IDs in the same transaction as the consumer’s own side effect, and skip IDs it has already seen.

That’s the same idea as idempotency keys for APIs, applied to messages — see Idempotency keys: making retries safe for money.

Operating it

• Clean up. Delete published rows older than a few days with a scheduled job, or partition the table by day and drop old partitions.
• Alert on the oldest unpublished row. now() - min(created_at) WHERE published_at IS NULL is the single best health metric: if it grows, events are stuck.
• Keep payloads self-contained. Put the data consumers need in the event, rather than making every consumer call back to your API — otherwise you’ve rebuilt the coupling the events were meant to remove.

Takeaways

• “Commit, then publish” is a dual write, and it loses events on crashes.
• Write the event into an outbox table in the same transaction; publish it afterwards.
• Relay by polling with SKIP LOCKED to start; move to CDC when latency or load demands it.
• The outbox is at-least-once: consumers must de-duplicate.