Idempotency keys: making retries safe for money

A request that times out may still have succeeded. That single fact is why wallet and refund systems — I’ve built both — have to be designed around retries from day one.

Here is the sequence. A client calls POST /refunds. The server debits the merchant, credits the customer, commits — and then the response is lost: a load balancer timeout, a pod restart, a mobile network drop. The client sees an error and does the reasonable thing: it retries. The server, seeing what looks like a new request, refunds again.

Retries are not the bug. Retries are essential; networks fail all the time. The bug is an API that can’t tell a retry from a new request.

The pattern

An idempotency key is a unique value the client generates once per operation — not per HTTP attempt — and sends with every attempt:

POST /refunds
Idempotency-Key: 6f1c2a9e-7d3b-4f0a-9a51-2c8e0b7d4e11
Content-Type: application/json

{"orderId": "A-1042", "amount": 1499}

The server promises: for a given key, the operation happens at most once, and every attempt gets the same response. Stripe popularised this design for its public API, and it applies to any endpoint that moves money, sends a message or creates something a user would notice twice.

The table

CREATE TABLE idempotency_keys (
    key            TEXT        NOT NULL,
    client_id      TEXT        NOT NULL,
    request_hash   TEXT        NOT NULL,          -- SHA-256 of method + path + body
    status         TEXT        NOT NULL,          -- IN_PROGRESS | COMPLETED
    response_code  INT,
    response_body  JSONB,
    created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (client_id, key)
);

Scope keys per client: two different clients may well generate the same string, and that must not leak one client’s response to the other.

The flow, with the edge cases

@PostMapping("/refunds")
public ResponseEntity<RefundResponse> refund(@RequestHeader("Idempotency-Key") String key,
                                             @RequestBody RefundRequest req,
                                             Principal client) {
    String hash = sha256(req);

    // 1. Reserve the key BEFORE doing any work. The primary key makes this atomic.
    int inserted = jdbc.update("""
        INSERT INTO idempotency_keys (key, client_id, request_hash, status)
        VALUES (?, ?, ?, 'IN_PROGRESS')
        ON CONFLICT (client_id, key) DO NOTHING
        """, key, client.getName(), hash);

    if (inserted == 0) {
        StoredKey existing = keys.find(client.getName(), key);
        if (!existing.requestHash().equals(hash)) {
            return ResponseEntity.unprocessableEntity().build();   // same key, different body
        }
        if (existing.status() == Status.IN_PROGRESS) {
            return ResponseEntity.status(409).build();              // concurrent retry: try again later
        }
        return existing.toResponse();                               // replay the stored result
    }

    // 2. Do the work and record the result in ONE transaction.
    return tx.execute(status -> {
        RefundResponse result = refunds.issue(req);
        keys.complete(client.getName(), key, 201, result);
        return ResponseEntity.status(201).body(result);
    });
}

Each branch exists because of a real failure mode:

Reserve before you work. If you check for the key, do the refund, and only then insert the key, two retries arriving 5 ms apart both pass the check. The INSERT ... ON CONFLICT DO NOTHING against the primary key is the lock: exactly one attempt wins the reservation.

Same key, different body. A client bug that reuses a key for a different refund must not get the first refund’s response — that would silently hide the second operation. Hash the request and reject mismatches with a 422.

In progress. A retry that arrives while the first attempt is still running gets a 409. The client backs off and retries; by then the first attempt has finished and the retry gets the stored response.

Work and result commit together. If the refund commits but storing the response doesn’t, the next retry finds an IN_PROGRESS row forever. Keep the business write and the key update in the same database transaction.

When the work isn’t in your database

The single-transaction trick only works when the side effect lives in the same database as the key table. When step 2 calls an external payment provider, you can’t roll that back. Two things help:

1. Pass the key downstream. Most payment providers accept their own idempotency key. Derive it from yours (for example refund:{clientId}:{key}), so a retry after a crash asks the provider for the same operation instead of a new one.
2. Recover stuck rows. A process can die between the provider call and the commit. A sweeper that finds rows stuck IN_PROGRESS for longer than your longest timeout can query the provider by the derived key and complete or release the row.

How long to keep keys

Keep keys at least as long as any client will retry, plus a margin. Stripe keeps them for 24 hours; for internal services where queues can redeliver messages days later, I’d lean longer. Delete old rows with a scheduled job — not inside the request path.

Where else this applies

The same idea shows up whenever “at least once” delivery meets “exactly once” side effects:

• Queue consumers. Kafka and SQS redeliver. Use the message ID (or a business ID) as the key and record it in the same transaction as the effect.
• Webhooks. Providers retry webhooks; store the event ID before acting on it.
• Scheduled jobs. A job that runs twice after a deploy should find its own key and stop.

Takeaways

• Retries are necessary; duplicate side effects are optional.
• The key belongs to the operation, generated by the client once and reused on every attempt.
• Reserve the key atomically before the work, store the response with the work, and replay it on retry.
• Reject a reused key with a different body, and pass keys to every downstream system that moves money.