Servers Dream in Binary, Monkeys Throw Exceptions

Distributed systems unravel when curiosity meets entropy. The architecture diagram shows boxes and arrows, and the arrows are the honest part — but they are drawn as though they either work or do not. The interesting failures live in the third state the diagram has no notation for: the arrow that works slowly, intermittently, or for some callers and not others.

Partial failure is the whole subject

A single machine has a comfortable property: it is either up or down, and when it goes down, everything on it goes down together. That is a brutal failure mode but it is a legible one. Recovery is a known procedure.

Distributed systems trade that for something subtler. A dependency can be reachable but slow. It can answer some requests and time out on others. It can be healthy from one availability zone and unreachable from another. It can return a response that is syntactically fine and semantically stale. None of these register as "down" on a dashboard built around a binary, which is why the first thirty minutes of a great many incidents are spent establishing that something is wrong at all.

The useful mental shift is to stop treating a remote call as a function call that occasionally fails, and start treating it as a message to a system that may or may not be listening, may or may not act, and may or may not tell you which. Everything unpleasant about distributed systems follows from taking that seriously — and most of the bugs follow from not.

The retry that makes it worse

Retries are the first tool everyone reaches for and the one most likely to convert a degradation into an outage.

The mechanism is not subtle once you see it. A dependency slows down. Callers time out and retry. The retries are additional load on a service that is already struggling, so it slows further, so more calls time out, so more retries arrive. The system has discovered positive feedback. Nobody designed this; it emerged from a sensible-looking default applied at several layers independently, each unaware of the others.

The compounding is what catches people. A retry policy at the client, another in the service mesh, and another in the SDK the client happens to use do not add — they multiply. Three layers each retrying three times is not three attempts, it is twenty-seven, and the service on the receiving end has no way to tell that twenty-six of them are echoes of the same original request.

The countermeasures are well understood and inconsistently applied: cap the total attempt budget across layers rather than per layer, add jitter so the retries do not arrive as a synchronised wave, and — the one people skip — make retries conditional on the failure being plausibly transient. A retry on a timeout is a reasonable bet. A retry on a 400 is just asking the same wrong question louder.

Timeouts are a design decision pretending to be a config value

Every remote call has a timeout. Most of them have whatever timeout the library shipped with, which was chosen by someone who had never seen your system and was optimising for not surprising a newcomer.

A timeout is really a statement about what you will do with the time you save, and it composes badly. If a caller's timeout is shorter than its dependency's, the caller gives up while work is still in flight — the work completes, consumes resources, and produces a result nobody is waiting for. If the caller's timeout is longer, the caller holds a connection open through the entire failure of something downstream, and the pool exhausts.

Neither of these is exotic. Both are the default outcome when timeouts are set independently at each layer by whoever wrote that layer. Getting them right means treating the timeout budget as a property of the whole call chain rather than of each hop, which requires somebody to own the chain — and call chains are precisely the thing that ownership boundaries cut across.

The rituals

Every team has debugging rituals, and it is worth being honest that some of them are cargo cult. Restarting the service does often help, but it helps by discarding accumulated state, and if you never ask which state, you will restart it again next week. Scaling out does often help, but if the bottleneck is a shared dependency, adding callers makes it worse in a way that is indistinguishable from making it better for about ninety seconds.

The distinguishing question for any ritual is whether performing it teaches you anything when it fails. "Restart it" teaches you nothing either way. "Drain one instance and see whether the error rate on the remaining ones changes" costs about the same and has an interpretable outcome in both directions.

That is really the whole discipline. Not fewer rituals — rituals are how tired people at three in the morning avoid having to think from first principles, and that is a legitimate function. Just rituals that are also experiments, so that the bad night produces knowledge as a side effect rather than only a restored service and a shrug.

Duct tape and optimism

Infrastructure is held together by duct tape and lots of optimism, and this is not a criticism. The duct tape is usually load-bearing for a good reason: it was applied during an incident by someone who understood the problem better than the original design did, and it has been holding ever since.

The failure is not having duct tape. The failure is not writing down which pieces are structural. Every system accumulates fixes whose rationale lives in one person's memory, and the entropy that unravels distributed systems is mostly organisational — the slow leak of the reasons things are the way they are. The comment explaining why the timeout is 800ms and not 1000 is worth more than the constant it annotates.