The DevOps tax: the real cost of self-hosting AI isn't the GPU.
The line item nobody budgets is engineering time: driver fights, CUDA versions, serving-stack tuning, Kubernetes. Why the GPU hourly rate was never the number that mattered.
In this article 2 sections
- The biggest hidden cost in self-hosted AI is engineering time, not the GPU hourly rate.
- Driver and CUDA compatibility, quantization choices, serving-stack tuning, and Kubernetes eat weeks before the first token is served.
- A raw GPU cloud hands you compute and a bill; the wiring-up is silently your payroll.
- The managed alternative: your team calls an endpoint, our engineers keep it running. You don't hire an MLOps engineer, you get ours.
Price a GPU server and you get a clean number. Price what it takes to turn that server into a production inference endpoint and the number goes soft: a driver version that disagrees with the CUDA toolkit, a serving framework that wants a different kernel, a quantization that turns out to need more VRAM than the blog post said, a Kubernetes layer the workload did not need. None of it appears on the invoice. All of it appears on your payroll.
The tax, line by line
Here is what sits between a card in a rack and an endpoint your application can trust, in the order it arrives. Every line is engineering time, and every line recurs: a stack that worked on the day it was installed has to keep working through every update that follows.
Burn-in
Prove the card and the host are healthy before anything depends on them: sustained load, thermals, memory, power. A card that throttles under its first real batch is a problem you want to find on day one, not on launch day.
Driver and toolkit alignment
Kernel, driver, CUDA toolkit, container runtime and serving framework all have to agree with each other. They agree on the day you pin them. Every update to any one of them reopens the question.
Model packaging
Weights, tokenizer, quantization and the VRAM budget that follows from them. The choice of quantization decides whether the model fits, how fast it answers and how much quality it gives up, and the published numbers are a starting point, not a guarantee for your hardware.
Serving-stack tuning
Batching, context length, KV-cache sizing and concurrency, tuned to your latency target rather than the framework's defaults. This is where most of the throughput lives, and it is work nobody sees when it is done well.
Endpoint plumbing
TLS, authentication, a private network path to the application that calls it, and the decision about what is reachable from where. An inference endpoint that is reachable from the public internet is a liability before it is a feature.
Observability
Card health, VRAM pressure, queue depth and latency percentiles, with someone actually watching them. A GPU that is quietly running at half speed looks exactly like a healthy one on a dashboard that only checks whether the process is up.
Updates, with a way back
Model, driver and stack updates applied on a schedule, tested before they land, with a rollback that has been exercised. Skipping updates buys quiet for a while and then a very loud week.
Load test
Prove the endpoint holds your concurrency at your latency target before anything depends on it. The first time the endpoint sees real traffic should not be the first time anybody has measured it.
On-call
When it breaks at 3 a.m., someone has to own it. Not on a rota in a spreadsheet: an engineer who knows the stack, awake, with access.
Which lines disappear when the people who racked the card also deploy the stack
Every one of them except two. You still choose the model, and you still write the application that calls it. Burn-in, version pinning, serving-stack tuning, the private endpoint, the monitoring, the updates, the load test and the on-call stay with the people who own the hardware and already carry them for every other machine in the room. We run inference on our own GPUs, so the list comes from practice.
Not a card by the hour. A private endpoint that answers at the latency you agreed, on a single-tenant machine, kept current by engineers who know the stack because they built it. The hourly rate is the smallest number on the page.
Straight answers
Can't we just use the vendor's install guide?+
The guides assume one card, one driver, one framework version, frozen in time. Production means the combination keeps working through updates, under load, at your latency target. That maintenance is the tax; the install was never the hard part.
Does managed mean we lose root?+
No. The machine is single-tenant and yours; how much of the stack we carry is agreed per customer: some take the keys and run everything, others hand us the serving layer and keep their application.
What does your team actually do ongoing?+
Version-lock drivers and CUDA, deploy and tune the serving stack, monitor the card and the endpoint 24/7, and apply model and stack updates on a schedule you approve. When it breaks, we fix it, at any hour.
Bring us the model you want to run
We size the GPU, deploy the stack, and hand you a private endpoint. The DevOps tax stays with us.
See GPU ServersYou’ve reached the end.
Put the reading to work
What are
you running?
Tell us about the workload, traffic and recovery requirements. Start with the server builder, or talk through the details with an engineer.
Written by the engineers who build and run the infrastructure it describes.Talk to an engineer