The Dual Write Problem, and Everything It Forces You to Build

Two lines of code, and most of distributed architecture exists because of them:

db.save(order);
broker.publish(new OrderPlaced(order.id));

There is no transaction around that pair. There cannot be: the database and the broker are separate systems with separate commit protocols. So the pair is not atomic, and the useful question is not whether it breaks but which way it breaks.

Everything below — outbox tables, idempotent consumers, sagas, backoff with jitter, schema registries — is downstream of those two lines. It is worth seeing the chain as a chain rather than as seven unrelated patterns to memorise.

Four ways to fail, not two

People usually picture one failure here. There are four, and they need different answers.

The commit succeeds and the publish fails. The order exists; nobody is told. Shipping never hears about it, the confirmation email never goes out, and your database and your downstream services now disagree permanently. This is the famous one.

The publish succeeds and the commit fails. Worse, and much less discussed. You have announced an order that does not exist. Downstream services start processing a phantom. Every consumer that trusted you now holds data with no source of truth behind it.

The publish succeeds and the acknowledgement is lost. The broker took the message; the network dropped the ack on the way back. Your client retries because it must — from where it sits, this is indistinguishable from case one — and the message is now in the log twice.

The process dies between the two calls. No log line, no exception, no stack trace. You find out from a support ticket three days later.

Note that swapping the order of the two statements does not fix anything. It only changes which of the first two failures you get. There is no ordering of a non-atomic pair that makes it atomic.

Why two-phase commit is not the answer

The textbook fix is a distributed transaction: XA, two-phase commit, a coordinator that makes both systems agree. In practice this is rarely the road taken, for reasons worth being specific about.

Kafka does not support XA at all, so for the most common broker in this space the option simply does not exist. Where XA is supported, the coordinator becomes a component whose failure blocks participants holding locks — during the window between prepare and commit, resources stay locked and the participants cannot decide alone. That window is exactly when you least want a coordinator outage. Holding database locks for the duration of a network round trip to a broker also puts your throughput ceiling in the hands of the slowest participant.

None of this means 2PC is wrong everywhere. It means the industry chose a different trade: keep one atomic write to one system, and derive everything else from it.

Transactional outbox

The insight is small. You cannot make two systems commit atomically, so stop trying to write to two systems. Write to one — the database — and make the message part of that same write.

   ┌──────────────────────────────────────────────┐
   │  ONE database transaction                    │
   │                                              │
   │   INSERT INTO orders  ...                    │
   │   INSERT INTO outbox  ...   ← the event      │
   │                                              │
   │  COMMIT  ── both, or neither                 │
   └──────────────────────────────────────────────┘
                      │
                      │  a separate process reads the outbox
                      ▼
              ┌───────────────┐
              │  Relay        │ ── publish ──▶  Broker
              │  (poll / CDC) │ ◀── ack
              └───────────────┘
                      │
                      ▼
              mark row as published

The business row and the event row land in the same commit, so the two can never disagree. Either the order exists and the event is queued, or neither happened. The first two failure modes are gone by construction — not handled, not retried, gone.

A minimal schema:

CREATE TABLE outbox (
  id             BIGSERIAL PRIMARY KEY,
  aggregate_type TEXT        NOT NULL,
  aggregate_id   TEXT        NOT NULL,
  event_type     TEXT        NOT NULL,
  payload        JSONB       NOT NULL,
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at   TIMESTAMPTZ
);

CREATE INDEX outbox_unpublished
  ON outbox (id) WHERE published_at IS NULL;

The partial index matters more than it looks. The table is mostly published rows and the relay only ever wants the unpublished tail, so indexing the whole table wastes most of the index. WHERE published_at IS NULL keeps it roughly the size of your backlog instead of the size of your history.

Getting the events out

Two ways to move rows from the table to the broker.

Polling. A loop selects unpublished rows, publishes them, marks them. It is easy to build, easy to reason about, and easy to debug — the queue is a table you can query. The cost is polling latency and constant load on the database.

If you run more than one relay instance, you need the rows claimed exactly once:

SELECT * FROM outbox
 WHERE published_at IS NULL
 ORDER BY id
 LIMIT 100
   FOR UPDATE SKIP LOCKED;

SKIP LOCKED is what makes several relays cooperate instead of colliding — each one takes rows the others have not locked, with no coordination and no waiting. Without it, instance two blocks behind instance one and you have bought concurrency you cannot use.

Change data capture. Instead of polling, tail the database's replication log — Debezium is the common implementation — and turn inserts into the outbox table into broker messages. Lower latency, no polling load, and it cannot miss a committed row because the log is the commit. The cost is real operational weight: another distributed system to run, with its own failure modes and its own lag to monitor.

Start with polling. Move to CDC when polling latency or database load actually becomes the constraint, not before.

Ordering, and the thing the relay cannot promise

If consumers care about the order of events for one entity — and they usually do, because OrderCancelled before OrderPlaced is nonsense — publish in id order and partition by aggregate_id so all events for one order land in the same partition. Global ordering across all orders is almost never needed and is very expensive to get; per-aggregate ordering is nearly always what people actually mean.

Now the important part. The relay can publish a message and die before marking the row as published. On restart, it publishes that message again. This is not a bug you can engineer away — it is the same lost-acknowledgement problem from case three, moved one layer down.

The outbox gives you at-least-once delivery. It cannot give you less. That fact is what the next section is about.

Finally, prune the table. Published rows are dead weight and an outbox that grows forever will eventually be the reason your database runs out of disk. Delete or partition-drop rows older than your replay window.

At-least-once, exactly-once, and what is actually achievable

Three delivery guarantees get named, and one of them is mostly a marketing term.

At-most-once: fire and forget. Fast, and it loses messages. Fine for a metrics ping, wrong for an order.

At-least-once: retry until acknowledged. Never loses a message, sometimes delivers it twice. This is what an outbox gives you, and what almost every real system runs on.

Exactly-once: delivered precisely one time. In the general case, across two independent systems, this is not achievable — it reduces to the Two Generals Problem. The sender cannot distinguish "receiver never got it" from "receiver got it and the ack was lost", and no number of extra messages resolves that.

Kafka's exactly-once semantics are real, and it is worth being precise about their scope, because the name invites over-reading. Kafka can make a consume-transform-produce cycle atomic within Kafka: read from a topic, write to another topic, commit the offset, all or nothing. The moment your consumer also writes to Postgres or calls a payment API, you are back to a dual write and Kafka's transaction does not cover it.

So the goal is not exactly-once delivery. It is exactly-once effect: the message may arrive many times, and the result must be the same as if it arrived once. That is idempotency, and it belongs to the consumer.

Making a consumer idempotent

Some operations are naturally idempotent. UPDATE orders SET status = 'PAID' WHERE id = ? can run a hundred times with one outcome. UPDATE accounts SET balance = balance + 100 cannot — that is a hundred outcomes.

Where the operation is not naturally idempotent, deduplicate explicitly:

BEGIN;
  INSERT INTO processed_messages (message_id) VALUES (?);
  -- unique violation here means: already handled, roll back and ack
  UPDATE accounts SET balance = balance + 100 WHERE id = ?;
COMMIT;

The two critical details are easy to get wrong. The dedup insert and the business effect must be in the same transaction — if you record the message id in a separate transaction, or in Redis, you have reinvented the dual write one level down. And the id must come from the producer, carried in the message, not generated on receipt. A broker redelivery must carry the same id it carried the first time.

Producer-side event ids want to be unique and, ideally, time-ordered so the dedup table's index stays healthy — the same argument that applies to primary keys. Generate a few if you want to see the difference, and UUID v4, v7 or ULID covers why the choice affects insert performance more than collision risk.

When there is genuinely no natural id — an event from a system you do not control — a content hash of the payload's stable fields is a workable fallback. It is weaker: two legitimately identical events become one. Use it only when the alternative is nothing.

The dedup table also needs a retention policy, bounded by how far back your broker can redeliver. Keeping ids forever is a slow leak; keeping them for less than the retention window is a correctness bug.

Sagas: transactions that span services

Idempotency handles one message. A business process spans several services — reserve stock, charge the card, book the courier — and a distributed transaction across all of them is exactly what you gave up. A saga is the replacement: a sequence of local transactions, each publishing an event or command that triggers the next, and each with a compensating action for undoing it.

Choreography

No central coordinator. Each service listens for events and reacts.

OrderPlaced ──▶ Payments  ──▶ PaymentCaptured ──▶ Inventory
                                                     │
                                                     ▼
                                              StockReserved ──▶ Shipping

The appeal is low coupling: adding a fraud check means subscribing to an event, with no existing service redeployed. This works well while the flow is short.

It degrades in a specific way. The process exists in no single place, so answering "what happens when an order is placed?" requires reading every service that subscribes. Debugging a stuck order means tracing events across service boundaries. And when steps grow past a handful, the event graph develops cycles that nobody designed and nobody can see.

Orchestration

A coordinator owns the flow and issues commands.

        ┌──────────────────┐
        │ OrderSaga        │
        └──────────────────┘
           │    │    │
     ┌─────┘    │    └─────┐
     ▼          ▼          ▼
  Payments  Inventory  Shipping

The flow is now a readable, unit-testable object. Timeouts have an obvious owner — the orchestrator knows a step was issued and not answered. "Where is this order?" has one answer in one place.

The cost is a component to build and operate, and a standing temptation for it to accumulate business logic that belongs in the services it calls. An orchestrator that grows decision rules for every participant has become the distributed monolith the architecture was meant to avoid.

A workable rule: choreography for two or three steps with no interesting failure handling; orchestration once the flow is long enough that someone will have to draw it on a whiteboard. Long-lived processes with timeouts and human approval steps are orchestration territory without argument.

Compensation is not rollback

This is where sagas most often go wrong in practice. A database rollback erases history. A compensation is a new action that semantically reverses a previous one, and the original stays visible.

Refunding a charge is not un-charging it. The customer sees both lines on the statement. Releasing reserved stock is not un-reserving it — another order may have taken that unit in between. Some steps have no compensation at all: an email that has been sent has been sent, and the best available compensation is a second email that says the first one was wrong.

That last category drives the design. Order the saga so that irreversible steps come last, after the point where success is nearly certain. The step past which you can only go forward is often called the pivot. Everything before it must be compensatable; everything after it must be retriable until it succeeds. Putting "send confirmation email" before "charge card" is a design error, not a bug in the code.

Explaining eventual consistency to people who do not write code

This is normally the skipped section, and it causes more project damage than any of the technical material above.

Start by not saying "eventual consistency". The phrase is heard as "sometimes wrong", and the argument you get back is "then make it not wrong", which is not a negotiation you can win on technical grounds.

Describe the observable behaviour instead. Not the read model is asynchronously projected, but: "when a customer places an order, the order page shows it immediately. The warehouse dashboard shows it within about two seconds. During those two seconds the dashboard total is the old number."

That framing does three things. It is concrete, so people can picture it. It is falsifiable, so it can be measured. And it turns the design into a business decision — the acceptable window and what the user sees during it are product questions, and stakeholders are perfectly capable of answering them once asked in their own terms.

Useful questions for that conversation:

• How long is acceptable here? Two seconds and two minutes are very different systems; people will often accept far more than engineers assume.
• What should the screen show meanwhile? A stale number with no explanation is the worst option. "Processing" is almost always fine.
• Who must never see stale data? Usually a much shorter list than "everyone" — and the honest answer is often only the person who just acted.

That last point has a name worth knowing: read-your-own-writes. Users forgive a colleague's change taking a few seconds to appear. They do not forgive their own change vanishing, because they read that as data loss. Routing a user's reads to the primary briefly after they write, or holding the optimistic value on screen, buys most of the perceived consistency for a fraction of the cost of real synchronous consistency.

On analogies, one caution. Bank transfers are the popular example and they mislead, because people believe banks are consistent and the delay is bureaucratic. Package tracking works better: everyone has seen a parcel that is physically moving before the tracking page admits it, and nobody concludes the parcel is lost.

Resilience: the four patterns, in the order they matter

Distributed calls fail. These four are not a menu — they compose, and the order matters.

Timeouts come first, and they are not optional. A call with no timeout is a resource leak with a delay fuse: threads and connections accumulate until the pool is exhausted, and then a slow dependency has taken down a service that was otherwise healthy. Every timeout must also fit inside its caller's budget. If your API must answer in three seconds and it makes two sequential calls with five-second timeouts each, the timeouts are decoration.

Retries, but only for the right failures. Retry transient errors — a timeout, a 503, a connection reset. Do not retry a 400; the request will be just as invalid the second time. And do not retry a non-idempotent write unless the receiver deduplicates, because retrying a charge is how one payment becomes three. Retries and idempotency are the same design decision viewed from two ends.

Exponential backoff with jitter. Backoff is intuitive: wait longer after each failure — 100ms, 200ms, 400ms, 800ms, capped. Jitter is the part people skip, and it is the part that matters at scale.

Consider a dependency that fails for every caller at once. Every client starts its backoff at the same instant, so every client retries at the same instant, and the dependency — which was trying to recover — receives a perfectly synchronised wave. It fails again, and now the clients are even more tightly synchronised than before. Deterministic backoff does not spread load; it organises a stampede.

Jitter breaks the synchronisation by randomising the delay:

delay = random_between(0, min(cap, base * 2 ** attempt))

That form — "full jitter" — samples the whole interval rather than a window around the target. It has the largest spreading effect, at the cost of occasionally retrying sooner than a strict backoff would. Variants keep a guaranteed minimum wait; the choice matters far less than having jitter at all. AWS's writing on this is the canonical reference and worth reading directly.

Circuit breaker. When a dependency is clearly down, retrying is worse than useless: it burns your threads and denies the dependency the quiet it needs to recover. A circuit breaker watches the failure rate and, past a threshold, opens — subsequent calls fail immediately without leaving the process. After a cooldown it goes half-open and lets a small number of trial calls through. Success closes it; failure re-opens it.

Trip on failure rate over a rolling window, not on a raw count, and require a minimum call volume before the rate means anything. A breaker that trips on "five failures" will trip on five failures out of five million calls, which is noise. The fallback behind an open breaker is a product decision: cached data, a degraded response, or a clean error — but decided deliberately, not whatever the exception handler happens to do.

Bulkhead. Named after ship compartments: a hull breach floods one compartment, not the vessel. Give each dependency its own bounded pool of threads or connections, so a slow one exhausts its own allocation and nothing else. Without bulkheads, one degraded downstream can consume every thread in a shared pool, and a service with nine healthy dependencies and one sick one goes down entirely.

Composed, roughly outermost to innermost: bulkhead limits how much of your capacity this dependency may ever consume; the circuit breaker decides whether to attempt the call; retries handle transient failure; the timeout bounds each individual attempt. Add a retry budget — a cap on retries as a fraction of traffic — if you run at a scale where retries themselves can become the load.

Versioning event contracts

The last piece, and the one that decides whether any of this survives two years.

A REST endpoint has consumers you can enumerate and, eventually, drain. A published event has consumers you do not know about, and — if your broker retains history — old versions of the event that live in the log and will be read again on every replay. You cannot migrate data that has already been written. Consumers must handle every version you have ever emitted.

Two compatibility directions are worth keeping straight:

• Backward compatible: a new consumer can read old events. This is what replay needs.
• Forward compatible: an old consumer can read new events. This is what lets you deploy producers and consumers independently.

You want both, which in practice means one rule: only add optional fields.

• Adding an optional field with a sensible default is safe. Old consumers ignore it; new consumers cope with its absence.
• Removing a field breaks every consumer still reading it.
• Renaming is removing and adding at once, so it breaks the same way.
• Changing a type breaks deserialisation, and the failure is often ugly and late.
• Redefining what a field means while keeping its name and type is the worst of all, because nothing fails. amount silently changing from cents to currency units passes every schema check and corrupts data quietly for as long as it takes someone to notice.

Enforce this mechanically rather than by review. A schema registry — Avro, Protobuf or JSON Schema — can check a proposed schema against the compatibility rule and reject the change in CI, before it reaches a topic. Anything left to human vigilance across many teams will eventually be missed.

When a genuine breaking change is unavoidable, do not mutate the existing event. Publish a new event type or a new topic, have the producer emit both for a migration window, move consumers across, and retire the old one once its subscriber list is empty. The version belongs in the event type name where a consumer cannot miss it — OrderPlaced.v2 — rather than only in a field that a deserialiser will happily ignore.

Two practices make this cheaper. Keep event payloads small and explicit; an event that dumps an entire entity turns every internal field into part of your public contract. And when a change does land, diff a real payload before and after rather than reasoning about the schema in the abstract — the field you forgot is usually visible immediately and invisible on paper.

The chain

Read as a list, these are seven patterns. Read as a chain, they are one problem and its consequences:

You cannot atomically write to two systems, so you write the event into your database in the same transaction and relay it — the outbox. The relay can crash after publishing, so delivery is at-least-once and consumers must be idempotent. Idempotency covers one message but not a multi-service workflow, so you need sagas, and sagas mean compensation rather than rollback. All of this makes the system eventually consistent, so the delay becomes a product decision that has to be stated in observable terms. The calls between services fail, so timeouts, backoff with jitter, circuit breakers and bulkheads. And every event is a contract you cannot recall, so versioning is additive and enforced by a machine.

Skipping a link does not remove it. It relocates it to an incident.