Retries are a load multiplier, not a reliability fix

The arithmetic of the second try

The standard justification for retries is simple: if a request fails due to a transient network glitch, sending it again will likely succeed. This logic holds true for a single, isolated client talking to a healthy service. In that narrow scenario, the retry is a corrective action that restores the intended state. However, distributed systems rarely operate in isolation. When you add a retry policy, you are not just fixing a failed transaction; you are introducing a new source of traffic that is conditional on failure.

Consider a service that receives one hundred requests per second. If it is healthy, it processes one hundred requests. If it begins to fail, and every client is configured to retry failed calls once, the service now receives one hundred original requests plus one hundred retries. The load has doubled. The service, which was already struggling, is now under twice the pressure. This is not a theoretical edge case; it is the direct arithmetic consequence of retrying. The more aggressive the retry policy, the higher the multiplier. If a client is set to retry three times, a total failure of the downstream service results in four times the original request volume hitting the same bottleneck.

This dynamic is often overlooked because the retry code lives in the client, while the capacity constraints live in the server. The client sees a failure and interprets it as a temporary hiccup. The server sees the incoming traffic and interprets it as a demand for more resources. The two sides of the conversation are fundamentally misaligned. The client is trying to be helpful by persisting; the server is trying to stay alive by shedding load. The retry mechanism forces the server to choose between serving the new, redundant work or dropping the original request entirely. In most production environments, the server will drop requests, meaning the retry has consumed processing time and network bandwidth without delivering a single successful result.

Where independence breaks down

The reliability argument for retries rests on a statistical assumption: that failures are independent. If each request has a ten percent chance of failing, and those failures are random and uncorrelated, then retrying increases the probability of success. The gRPC documentation, for instance, describes retries as a way to overcome temporary issues like network glitches, implying that the fault is transient and specific to that single call path. This model works when the underlying infrastructure is stable and the failures are noise.

However, the failures that cause major outages are rarely independent. They are usually correlated. A database connection pool runs out of connections. A network partition isolates a data center. A garbage collection pause stalls the entire application. In these scenarios, the cause of the failure is systemic. Retrying a request during a network partition does not fix the partition. Retrying a request when the connection pool is exhausted does not create a new connection. It simply queues another request behind the ones already waiting. The failure rate does not drop to zero; it stays high, or gets higher, because the system is saturated.

This is where the concept of "second tries" becomes critical. Marc Brooker’s analysis of backoff and retries distinguishes between first tries and second tries. First tries are the original user requests. Second tries are the retries. When a system is overloaded, the problem is often not that there are too many first tries, but that there are too many second tries. If you have a bounded number of clients, backing off might help by delaying their next first try. But if you have an unbounded number of clients, or if the clients are short-lived and stateless, they do not know that others are backing off. They just see a failure and retry. The result is a flood of second tries that adds no value but consumes all available capacity. The independence assumption fails because the failures are not random; they are a symptom of the load that the retries are creating.

The cost of saturation

When a service becomes saturated, its behavior changes. It no longer processes requests at a constant rate. Instead, latency increases as queues grow. Eventually, requests time out. These timeouts are then interpreted by clients as failures, triggering retries. The retries add to the queue, increasing latency further, causing more timeouts, and triggering more retries. This is a positive feedback loop. The system is not just failing; it is actively degrading its own ability to recover.

The Google SRE book discusses handling overload and notes that modeling capacity as simple queries per second can be misleading because different requests have different resource costs. But even if all requests were identical, the retry mechanism changes the effective load. A service designed to handle one thousand requests per second is no longer handling one thousand requests per second when it is under retry pressure. It is handling one thousand plus the retry volume. The service may have the hardware to handle the original load, but it does not have the hardware to handle the original load plus the amplified retry load.

This is why "just add retries" is a dangerous default. It assumes that the service has spare capacity to absorb the extra load. In a well-designed system, there is some headroom. But in a saturated system, there is no headroom. The retries eat the headroom. The service, which might have recovered if left alone, is instead kept in a state of perpetual overload. The retries are not helping the service recover; they are preventing it from recovering. The cost of this is not just failed requests; it is the time it takes for the system to stabilize. Every second of retry amplification is a second that the system is further from a healthy state.

Bounding the damage

The solution is not to remove retries entirely. Retries are useful for truly transient faults. The solution is to bound the damage they can do. This requires mechanisms that limit the number of retries based on the current state of the system. A circuit breaker, for example, can stop sending retries if the failure rate exceeds a threshold. A token bucket can limit the rate of retries, ensuring that the retry traffic does not exceed a certain percentage of the total traffic.

Marc Brooker’s work on adaptive retries suggests using a token bucket where successful calls deposit tokens and failed calls consume them for retries. This creates a feedback loop: if the system is healthy, there are tokens available for retries. If the system is failing, tokens are consumed quickly and the bucket empties, stopping the retries. This prevents the retry storm from overwhelming the service. Similarly, retry throttling in gRPC allows the server to push back against the client, telling it to slow down.

The key is to recognize that retries are a load, not a fix. They are an additional resource consumption that must be managed like any other load. The reliability argument for retries only holds while failures are independent and the system has capacity to absorb the extra load. The moment those conditions fail, retries become the mechanism of the outage. Engineers who wrap a client in a retry helper must understand that they are not just adding resilience; they are adding a potential failure mode. The goal is to make the retry logic smart enough to know when to stop, not just aggressive enough to keep trying.