Retries without a retry storm: timeouts, backoff, jitter and circuit breakers

Retries are the cheapest reliability feature there is: a request fails because of a dropped packet or a pod restart, you try again, it works. Most transient failures are gone in milliseconds.

They’re also one of the most common ways a small incident becomes a large one. A database hiccups for a minute. Every caller retries. The callers of those callers retry too. The database — the one thing that needed less traffic to recover — now receives several times its normal load, and the one-minute blip becomes a long outage. This pattern is often called a retry storm.

Here are the four tools that keep retries on your side.

The arithmetic of retry amplification

Imagine a request that goes through three services before it reaches a database, and each service retries a failed call 3 times in total (1 attempt + 2 retries).

When the database is failing, one user request becomes 3 × 3 × 3 = 27 database calls. Add users refreshing the page and you’ve multiplied your peak load by an order of magnitude at the exact moment you can least afford it.

Tool 1: timeouts with a budget

A retry without a timeout is a request waiting forever, holding a thread and a connection. Every outbound call needs a timeout, and the timeouts need to fit together:

• The user-facing request has an overall budget — say 2 seconds.
• Each hop below gets less than its caller, minus the time already spent. If service A gives B 1.5 s, B can’t give C 2 s.
• A deadline can travel with the request (gRPC does this natively; with HTTP you can pass a header) so a service deep in the stack knows when nobody is waiting for its answer any more.

The common mistake is the default: many HTTP clients ship with very long or infinite timeouts. Set them explicitly, per dependency.

Tool 2: exponential backoff with jitter

Retrying immediately mostly hits the same failure. Retrying after a delay that grows — 100 ms, 200 ms, 400 ms — gives the dependency time to recover.

But backoff alone isn’t enough. If 10,000 clients all failed at 12:00:00.000 and all wait exactly 100 ms, they all retry at 12:00:00.100 — a synchronized wave. Jitter spreads them out by randomising each delay. AWS’s architecture blog has a well-known analysis showing that “full jitter” (a random delay between zero and the backoff ceiling) dramatically reduces both total calls and completion time compared with plain exponential backoff.

With Resilience4j in a Spring service:

RetryConfig config = RetryConfig.custom()
        .maxAttempts(3)                                   // 1 call + 2 retries
        .intervalFunction(IntervalFunction.ofExponentialRandomBackoff(
                Duration.ofMillis(100),                   // initial interval
                2.0,                                      // multiplier
                0.5))                                     // randomization factor (jitter)
        .retryOnException(e -> e instanceof IOException
                            || e instanceof HttpServerErrorException.ServiceUnavailable)
        .build();

Retry retry = Retry.of("inventory", config);
Supplier<Stock> call = Retry.decorateSupplier(retry, () -> inventoryClient.getStock(sku));

Note the retryOnException predicate: retry transient failures (I/O errors, 503s, 429s), never 400 Bad Request or 404 Not Found. Those will fail the same way every time.

Tool 3: retry in one place, and only what’s safe

Two rules remove most of the amplification:

1. Retry at one layer. Usually that’s the layer closest to the failing dependency, or the edge, but not both. If the service right above the database already retries with backoff, the services above it should fail fast and let their own callers decide.
2. Retry only idempotent operations. A GET is safe. A POST /payments is only safe to retry if it carries an idempotency key the server honours — otherwise a retry after a lost response charges twice. (I wrote about that pattern in Idempotency keys: making retries safe for money.)

A retry budget is a useful extra guard: allow retries to add at most, say, 10% on top of normal traffic. When the budget is exhausted, fail fast. Envoy and several service meshes support this directly.

Tool 4: circuit breakers

When a dependency is clearly down, even well-behaved retries are wasted work. A circuit breaker tracks recent failures and, past a threshold, “opens”: calls fail immediately without touching the network. After a cool-down, it lets a few trial calls through (“half-open”); if they succeed, it closes again.

CircuitBreakerConfig cb = CircuitBreakerConfig.custom()
        .slidingWindowSize(50)                            // look at the last 50 calls
        .failureRateThreshold(50)                         // open at 50% failures
        .slowCallDurationThreshold(Duration.ofSeconds(1))
        .slowCallRateThreshold(80)                        // slow calls count too
        .waitDurationInOpenState(Duration.ofSeconds(10))  // cool-down
        .permittedNumberOfCallsInHalfOpenState(5)
        .build();

Two details matter in practice:

• Count slow calls, not just errors. A dependency that answers in 9 seconds is worse than one that errors in 9 milliseconds — it holds your threads. Slow-call thresholds catch that.
• Have a fallback. When the breaker is open, return something useful: a cached value, a default, a degraded response (“prices updating”), or a clear error. “Open breaker → 500” protects the dependency but not your users.

Putting it together

The order of decorators matters. A sensible stack, from outside in:

1. Timeout for the whole operation (the budget).
2. Retry with exponential backoff and jitter, on transient errors only.
3. Circuit breaker, so retries stop as soon as the dependency is known to be down.
4. The actual call, with its own connect and read timeouts.

And watch it: retry counts, breaker state changes and timeout rates belong on the same dashboard as error rates. A rising retry rate is often the first sign of trouble, minutes before errors reach users.

Takeaways

• Retries multiply load across layers: count your layers before you add a retry.
• Always pair retries with timeouts, exponential backoff and jitter.
• Retry only transient errors, only idempotent operations, in one layer.
• Use circuit breakers with slow-call detection and a real fallback.