The control loop that thinks it is a safety net
The loop and its delay
Autoscaling is frequently presented as a replacement for capacity planning, a magical mechanism that ensures your system always has exactly the resources it needs. This view ignores the fundamental nature of the mechanism itself. Horizontal autoscaling is a control loop, and like all control loops, it operates on a timescale. It observes a metric, makes a decision, and executes a change. This process is not instantaneous. The Kubernetes documentation describes Horizontal Pod Autoscaling as a controller that adjusts the number of pods based on observed resource utilization. This observation and adjustment cycle takes time. The system must gather metrics, process them, create new pods, schedule them onto nodes, pull images, start containers, and wait for readiness probes to pass.
The delay is not a bug; it is a physical constraint of the infrastructure. During this window, the system is operating with the old capacity. If the load increase is gradual, the loop catches up. It scales out a little, the load per pod drops, and it scales out a bit more. This works for slow, predictable growth. However, the loop’s response time sets a hard limit on the rate of change it can handle. Any load that arrives faster than this loop can react will find the system under-provisioned. The autoscaler is not a buffer; it is a follower. It can only follow load that moves slower than it can move itself. The assumption that autoscaling removes the need for planning is true only for a very specific, slow-moving class of load, which is rarely the class of load that causes outages.
The two load shapes that outrun it
The load shapes that typically cause problems are precisely those that move faster than the autoscaling loop. The first is the sudden spike. A marketing email goes out, a celebrity tweets about your product, or a competitor’s site goes down. In these cases, the load does not ramp up gradually; it steps up almost instantly. The autoscaler sees the spike, but by the time it has created the new pods, the spike has already passed, or the system has already failed under the initial burst. The loop is too slow to prevent the initial overload. It may scale up afterwards, but the damage is done. The system has already returned errors or shed load.
The second shape is the sustained high load that exceeds the maximum scale-out limit. Most autoscalers have a maximum number of replicas. This is a safety valve to prevent runaway costs or resource exhaustion. If the load is high enough to require more replicas than this maximum, the autoscaler will scale to the limit and stop. The system is now at its maximum capacity, and the load is still higher. The autoscaler cannot do anything more. It is not a magic wand that can create infinite compute. It is a bounded controller. In this scenario, the system is overloaded, and the only option is to degrade service or drop requests. The autoscaler has done its job, but its job was to scale within limits, not to eliminate the need for capacity. These two load shapes, the sudden spike and the sustained overload, are the ones that most commonly cause incidents, and they are the ones that autoscaling is least equipped to handle.
The dependency that gets worse rather than better
The second major failure mode of autoscaling is its interaction with shared dependencies. Many services in a distributed system do not run in isolation. They depend on shared resources such as databases, message queues, or external APIs. When you scale out a service, you increase the number of clients making requests to that shared dependency. If the dependency is the bottleneck, scaling out the client service makes the problem worse. You are sending more traffic to a resource that is already saturated.
The Kubernetes documentation notes that autoscaling is based on resource utilization of the pods themselves. It does not consider the load on downstream dependencies. If your database is at capacity, and you scale out your application pods, you are simply adding more pressure to the database. The application pods may have plenty of CPU and memory, but they are waiting on the database. The autoscaler sees low CPU utilization on the pods and may not scale up, or if it does scale up, it makes the database contention worse. This is a fundamental mismatch. The autoscaler optimizes for the resource it can see, not the resource that is actually limiting the system. In such cases, the bottleneck is not the compute, but the shared dependency. Scaling out the compute tier does not solve the problem; it exacerbates it. The system needs to scale the dependency, or the application needs to be designed to handle the dependency’s limits gracefully.
Why capacity planning still matters
The conclusion is not that autoscaling is useless. It is a valuable tool for handling gradual, predictable load changes. It reduces the need for manual intervention in the normal case. However, it is not a substitute for capacity planning. Capacity planning is the process of determining how much resource you need to handle your expected load, including the unexpected. It involves understanding your load shapes, your dependencies, and your limits. It is the process of setting the minimum and maximum replicas for your autoscaler. It is the process of ensuring that your shared dependencies have enough capacity to handle the load from your scaled-out services.
The teams that stop doing capacity work when they turn on autoscaling are setting themselves up for failure. They are relying on a control loop to handle problems that are outside its design envelope. The autoscaler will handle the slow ramp-up. It will not handle the sudden spike, the sustained overload, or the shared bottleneck. These are still planning problems. They require human judgment, architectural decisions, and explicit capacity allocation. The autoscaler is a tool, not a strategy. It is a component of your capacity plan, not a replacement for it. The most reliable systems are those that use autoscaling for the easy cases, and have explicit, well-planned capacity for the hard ones. The control loop is a follower, not a leader. It can only follow the load if you have planned for the places where the load will go.