Half Code, Half Banana, Autonomous Deployments

Automation gone feral is not a malfunction. It is what automation does when it is maintained by accretion, which is how nearly all of it is maintained. Pipelines accumulate behaviour the way old houses accumulate wiring: every addition was reasonable at the time, nothing was ever removed, and now there is a switch in the hallway that nobody dares to flip.

The pipeline is a program nobody reviews

The code that ships your code is code. It has control flow, state, error handling, and dependencies. It is also, in most organisations, the least reviewed program in the building.

Application changes go through review, testing, and a staged rollout. Pipeline changes go through "the build passed". This asymmetry is understandable — the pipeline's output is the test — but it means the delivery system evolves under a much weaker selection pressure than the thing it delivers. Errors persist in it that would never survive a week in production code, because the only failure mode anyone checks for is "did it go green".

The result is a class of bug that is invisible by construction: the pipeline that produces a correct artifact for the wrong reason. A test suite that has been silently skipping a directory since a path changed. A cache key that no longer includes an input it depends on, so the build is reproducible only in the sense that it reliably produces yesterday's answer. Nothing alerts, because from the outside these look exactly like success.

Caches become state, and state becomes policy

Every non-trivial pipeline caches, because without caching the feedback loop gets long enough that people stop using it. And every cache is a small database with no schema, no migration story, and no owner.

The trouble starts when the cache key stops describing the cache contents. Someone adds a build input — an environment variable, a generated file, a dependency resolved at build time — and does not add it to the key. From that moment the pipeline is capable of returning a stale artifact, and it will do so intermittently, depending on which runner picks up the job and what that runner happens to have lying around.

This produces the most disorienting failure in delivery engineering: the build that fails on CI and passes locally, or the reverse, with no difference in the source. People respond by clearing the cache, which works, which teaches them that clearing the cache is the fix, which means the actual defect — a key that does not cover its inputs — is never found. "Clear the cache and re-run" graduates from a workaround into a documented step, and the pipeline has acquired a ritual.

Self-modifying delivery

Pipelines that update themselves are genuinely useful and genuinely a different category of risk. A job that bumps its own dependencies, regenerates its own config, or opens a merge request against its own definition is writing to the system that decides what gets shipped.

The specific hazard is that the review step and the thing being reviewed can end up on the same side of the trust boundary. If a pipeline runs configuration from the branch it is testing, then a change to that configuration is executed before anyone has approved it. This is a well-known supply chain concern and it has a well-known shape of answer — run untrusted branches with reduced privilege, keep credentials out of jobs triggered by changes that have not been reviewed, and treat the pipeline definition itself as a protected path requiring a different approval than ordinary code.

The reason it keeps happening anyway is that the safe configuration is less convenient in exactly the situations where convenience is most wanted: fixing the pipeline. Nobody wants to require two approvals on the change that unblocks the release. The exception gets made once, and it stays.

Release cycles governed by lunar phases

The joke about lunar phases lands because delivery schedules genuinely do acquire periodicity nobody chose. Scheduled jobs pile up at the top of the hour because that is what people type. Nightly builds contend with backup windows. Weekly cleanups run on Sunday, so Monday's first build is the one that discovers what the cleanup removed.

Every one of those is a coupling that exists in wall-clock time and appears nowhere in any dependency graph. The pipeline does not know about the backup job. The backup job does not know about the pipeline. They are related only by both being scheduled at a round number, by different people, at different times, for unrelated reasons — and the relationship only becomes visible during the failure it causes.

Staggering scheduled work is a small piece of hygiene with a return out of proportion to the effort. Not because any given collision is likely, but because collisions in time are the hardest class of coupling to reason about after the fact: they leave no trace in the code, they reproduce only at the original hour, and the person debugging at 09:15 has no reason to look at what runs at 09:00.

What tames it

Three things, in descending order of how much people enjoy hearing them.

Make the pipeline's inputs explicit, so that a cache key covers what the build actually reads and a stale result becomes impossible rather than unlikely. Make the pipeline's changes reviewable on the same terms as application changes, especially the ones that grant it more power. And periodically delete something — a job nobody can explain, a step whose purpose is folklore, a workaround for a bug that was fixed upstream two years ago.

The last one is the hardest, because every piece of accumulated pipeline weirdness was load-bearing once and nothing marks the moment it stopped being. But a delivery system that only ever grows is a delivery system whose behaviour is guaranteed to exceed anyone's understanding of it, and the point at which that happens is not announced either.