My GPU Filed for Remote Work
Notes from the Era of Elastic Compute
High-performance computing used to feel like engineering. You bought hardware, installed drivers, optimized kernels, and measured FLOPs with the quiet pride of someone who could explain cache hierarchies at parties. The machine was in the building. If it was slow, that was a fact about the machine, and the machine was yours to interrogate.
Then the accelerator moved out. It got a job somewhere else, in a building you have never visited, and now it sends you results on a schedule that depends on someone else's capacity planning. This is mostly an improvement. It is worth being precise about what kind of improvement it is, because the thing that changed is not really performance.
What actually moved
The pitch for renting compute is elasticity: you stop paying for idle silicon and start paying for work performed. That part is true and it is genuinely transformative for workloads that are bursty, which is most of them. Training runs are lumpy. Inference is diurnal. Buying for peak means buying a lot of machine that spends most of its life warm and unemployed.
But elasticity was never the only property you had when the hardware was local. You also had locality, and locality is the thing that quietly stops being true. Your data and your accelerator used to be separated by a PCIe bus. Now they are separated by an object store, a network, a scheduler, and a queue. None of those are slow in an absolute sense. All of them are slow compared to a bus, and — more importantly — all of them are variably slow, which is worse than being consistently slow because you cannot design around it.
The practical consequence is that the bottleneck moves and stops announcing itself. A kernel that was compute-bound on a local card becomes input-bound in a rented one, and the profiler will happily tell you the GPU is underutilised without telling you why. It is underutilised because it is waiting. It has become, functionally, a very expensive consumer of a very ordinary network.
The skill that atrophies
When you own the machine, optimisation is a closed problem. There is a fixed amount of hardware, a known memory hierarchy, and a direct relationship between the code you write and the time it takes. That relationship is legible. You can reason your way from a change to an outcome.
Rented compute breaks the legibility, not the performance. The same code on the same nominal hardware can differ run to run because of placement, because of a noisy neighbour, because the instance you got this morning is in a different rack to the one you got yesterday. Engineers respond to this the way people always respond to non-determinism: they stop trying to reason and start trying things. The optimisation loop degrades from analysis into search.
This is not a moral failing and it is not laziness. It is a rational response to an environment where the feedback signal has become noisy. But it does mean the discipline that made HPC feel like engineering — measure, hypothesise, change one thing, measure again — needs deliberate reconstruction. You have to build the determinism back yourself, by pinning what you can pin, by running the same shape of job enough times to see through the variance, and by being honest that a single benchmark run on rented hardware is an anecdote.
Cost is now a performance metric
The other thing that changed is that the units moved. Owned hardware has a cost that is fixed at purchase and then irrelevant; every optimisation after that is pure gain in time. Rented hardware has a cost that is a live function of how long your job runs, which means that for the first time, "make it faster" and "make it cheaper" are the same sentence.
Mostly this is healthy — it puts a number on inefficiency that finance can read, and it ends the argument about whether the optimisation was worth the engineer's week. But it also introduces a failure mode that owned hardware never had: the job that works fine and costs a fortune. Nothing alerts. Nothing pages. The pipeline is green, the results arrive, and the only symptom is a line item that grows quietly until somebody notices the shape of the graph a quarter later.
Owned hardware wasted your capacity. Rented hardware wastes your money, and money has no utilisation dashboard unless you build one. The instrumentation most teams carry over from the local era measures the wrong resource entirely.
What to keep
None of this is an argument for buying racks again. For most workloads the economics are not close, and the operational burden of owning accelerators — drivers, firmware, cooling, the long tail of hardware that fails in interesting ways — is a real cost that the on-premise nostalgia tends to forget.
It is an argument for noticing which properties you traded away, because they were never itemised on the invoice. You gave up locality, determinism, and a legible relationship between code and time. You got elasticity, someone else's operations team, and a bill that responds to your engineering. That is a good trade for most people most of the time.
It is only a bad trade when you keep optimising as though the accelerator is still in the building. It is not. It has a commute now, and the commute is the part you have to design around.