Why do naive retries make an outage worse, and how would you design a safe retry policy?

Naive retries multiply load exactly when a service is already struggling, and synchronized clients retry in lockstep waves. Use exponential backoff with full jitter, cap total attempts, and only retry idempotent operations on retryable errors. Add a client side retry budget so retries stay a small fraction of traffic, plus a circuit breaker to stop calls entirely when failure rates spike. Never retry independently at every layer of the stack.

The problem is that retries are extra load applied at the worst possible moment. If a service is failing at 50% and every client retries three times, you have just tripled the traffic hitting something that is already over capacity, and it never gets a chance to recover. Plain exponential backoff is not enough either, because all your clients failed at the same instant, so they all wake up at the same instant. Full jitter fixes that, so the sleep is a random value between zero and the current backoff ceiling. Beyond that I would only retry idempotent calls, only on 503, 429 and connection level errors, never on a 400, and I would enforce a retry budget so retries are capped at something like 10% of total requests. The one that bit us was layering. The SDK retried three times, the mesh retried twice, and the job runner retried again, which is eighteen attempts for one logical call.