A 70B model really does fit on a 48GB card, once you count what quantization costs.

There are two ways to fail at GPU sizing and most teams pick one. The first team asks what it costs to run a serious model, sees the price of an 80GB card, flinches, and shelves the whole project. The second team is thriftier: it crams the model into the cheapest GPU that technically loads it, ships, and loses answer quality it never measured, because nobody told it what the compression cost. Opposite failures, same cause. Nobody had the sizing conversation, because most of the industry treats it as a spec-sheet mystery instead of what it is: arithmetic, plus one honest trade-off.

The arithmetic anyone can check

Here is the rule of thumb, flagged as exactly that; architectures and serving stacks shift the constants, but the frame holds. A model parameter stored at 16-bit precision takes two bytes. At 8-bit, one byte. At 4-bit, half a byte. Multiply by the parameter count and you have the weights' footprint. Then leave room for everything serving adds: the engine itself, and the KV cache, which grows with context length and with every concurrent user. That last part is why "the weights technically fit" and "this serves production traffic" are two different sentences.

Run the 70B example in round numbers. At 16-bit, 70 billion parameters times two bytes is roughly 140GB before overhead: multi-GPU territory. At 8-bit, roughly 70GB: an 80GB card, comfortably, with room to breathe. At 4-bit, roughly 35 to 40GB plus overhead: a 48GB card, with space left to actually serve. One model, three precisions, three very different price tags. Which is why "what GPU do I need" has no answer until someone asks the follow-up: at what precision, for what workload.

What each card class honestly serves

The pairings below are the same ones the board on our GPU Servers page walks you through, stated in prose. They are honest in both directions: what fits, and what the fit costs.

What quantization actually costs

Quantization compresses the precision of the model's weights, and the quality cost is real, workload-dependent, and measurable. For a lot of production work, grounded question-answering over retrieved documents, classification, extraction, summarization, modern 4-bit quantization is hard to distinguish from full precision in blind checks on your own data. For quality-critical generation, long chains of reasoning, or work where the margins are already thin, the gap can matter. Anyone who tells you a universal percentage is making it up. The honest number comes from one place: your evals, on your tasks, run against both versions, compared.

That eval run is part of how we deploy, not an exercise left to the reader. The endpoint gets load-tested against your latency target, and the quantization decision gets made with the trade-off in front of you, because it is your quality bar and your money. We would rather show you the two outputs side by side than hide the compression inside a cheaper quote.

And the direction nobody expects a hardware seller to point: down. Sometimes the honest answer is not the 80GB card, and sometimes it is not the 70B model either. A well-chosen 24B to 32B model at full precision beats a heavily squeezed 70B on plenty of real task sets, and it costs less to run. We size from the workload, we show you the trade-off, and when the smaller machine covers the job we say so before the invoice, not after. It is the same reflex as the rest of the house: size down, not up.

“We pick the card from the model, not the model from the card.”