The 2 GB per billion parameters rule
A LLM parameter is nothing more than a number. The only question that sets its memory cost is: how many bytes do you store it on? Models are trained and shipped in 16-bit, the FP16 16-bit floating point. A 2-byte number storage format, the default precision for an LLM's weights and activations. Two variants coexist: FP16 (IEEE half precision) and BF16 (the same range as FP32, less fractional precision). It is the baseline whose footprint quantization reduces. format or its cousin BF16, so 2 bytes per parameter. The arithmetic is then direct: a 7-billion-parameter model weighs 7 × 2 = 14 billion bytes, or ~13 GiB of weights. That is all the famous “2 GB per billion” rule says: it is not an empirical approximation, it is the arithmetic of the format.
What makes the rule useful is that it carries over to any precision. Drop the weights to 8-bit and it is one byte per parameter, half. In 4-bit it is half a byte, a quarter. quantization Reducing the number of bits that encode each weight of a model (from 16 bits down to 8, 4, or fewer). It shrinks the memory footprint by the same factor, at the cost of a controlled accuracy loss, without changing the parameter count. does not change the number of parameters, only the number of bits that encode each one: it is a multiplication, not a rewrite of the model.
Three line items, not one
The VRAM consumed by a model that serves is not that of the weights file alone. Three line items add up, and it is their sum that must fit on the card:
- The weights: fixed, given by the rule above. It is the only line item most people compute, which is why many get it wrong.
- The KV cache: the memory of what the model has already read. It starts at zero and grows with every token generated, proportionally to the context length and to the number of requests served in parallel. It is the KV cache, not the weights, that saturates VRAM first on long contexts.
- The overhead: the CUDA Compute Unified Device Architecture. NVIDIA's GPU computing platform: language, compiler and libraries (cuBLAS, cuDNN). Its software ecosystem is the main lock-in against alternatives like ROCm; at runtime, its context also reserves an incompressible slice of VRAM. context, activation buffers, allocator fragmentation. Budget ~1 GB incompressible, plus a few percent of the weight volume.
This breakdown explains the classic trap: a model that “fits” in VRAM at load time overflows during use. The weights fit; the KV cache of a long context had not been budgeted.
The table: what the weights weigh
Before adding the rest, you need to know what the bare model costs. The table below applies the rule to the three most common sizes, in the three deployment precisions:
| Model | 16-bit (FP16) | 8-bit (INT8) | 4-bit |
|---|---|---|---|
| 7–8 B | ≈ 14–16 GB | ≈ 7–8 GB | ≈ 4–5 GB |
| 13 B | ≈ 26 GB | ≈ 13 GB | ≈ 8 GB |
| 70 B | ≈ 140 GB | ≈ 70 GB | ≈ 42 GB |
The 4-bit column deserves a caveat: a quantized format never drops exactly to half a byte per parameter. Schemes like GGUF GPT-Generated Unified Format. llama.cpp's file format that stores the tensors, their quantization types, the vocabulary and the model metadata in a single file. Its aligned layout lets it be read with mmap, so the model starts in a fraction of a second. Q4_K_M (the quantized weight file that llama.cpp loads) store, on top of the weights, per-block scale factors covering a handful of values: the real footprint lands around 4.8 bits per parameter, or ~0.6 byte. That is why a 70B “in 4-bit” (Q4_K_M) weighs ~42 GB and not 35: the difference is the quantization metadata, and it is not optional.
Which card for which model
In practice, available VRAM comes in tiers, those of the cards on the market. Setting the tiers against the models gives a direct decision grid:
| VRAM | Typical cards | What fits comfortably |
|---|---|---|
| 8 GB | RTX 4060, entry-level cards | 7–8 B in 4-bit |
| 12–16 GB | RTX 4070, RTX 4060 Ti 16G | 13 B in 4-bit, 7–8 B in 8-bit |
| 24 GB | RTX 3090, RTX 4090 | 32 B in 4-bit, 13 B in 8-bit |
| 32 GB | RTX 5090 | 32 B in 4-bit (comfortable); 70 B only in ~3-bit (very tight) |
| 48 GB | RTX A6000, 2× 24 GB | 70 B in 4-bit |
| 80 GB | H100, A100 80G | 70 B in 8-bit (moderate context) or 4-bit (long context) |
Two readings of this table are worth spelling out. First: the price jump between 24 and 80 GB does not reflect a proportional jump in raw capacity, but access to 70B models and long contexts, which is the whole stake of the RTX 5090 versus H100 matchup for local LLMs. Second: past 24 GB, the trade-off is no longer just “buy a bigger card” but “buy or rent”: a 70B served a few hours a day costs less in cloud GPU than in amortized hardware.
Reducing the VRAM you need
When the model you want does not fit, three levers exist, in decreasing order of payoff.
The first, and by far the most powerful, is weight quantization. Going from 16 to 4-bit divides the main line item by four, which fits a 70B in 4-bit (~42 GB) onto two consumer cards, where its 140 GB in FP16 force several datacenter GPUs. The quality loss of a well-calibrated Q4_K_M stays small on most workloads; it is the best-established trade-off in local inference.
The second is KV cache management. Reducing the maximum context length, quantizing the cache to FP8 An 8-bit floating-point format. A versatile working format for inference (and training) on recent GPUs. It halves the memory footprint and bandwidth needed versus FP16, for a marginal accuracy loss on most models. (one byte per value instead of two, so double the capacity), or enabling prefix caching frees exactly the VRAM that long contexts were claiming. On a high-context workload, this lever returns more usable memory than a card upgrade.
The third is offloading: moving part of the layers to CPU RAM, as llama.cpp allows. It unblocks running a model too big for VRAM, but at a steep price: every layer in RAM is paid for in round trips over the PCIe Peripheral Component Interconnect Express. The bus linking the GPU to the CPU and RAM. Its bandwidth (a few tens of GB/s) is one to two orders of magnitude below the VRAM's, which makes the host↔GPU transfer a bottleneck as soon as data is moved off the card. bus, and throughput collapses. It is a last-resort fix, not a sizing strategy.
Conclusion
The question “how much VRAM for an LLM?” has an arithmetic answer for the weights, two gigabytes per billion divided by the precision, but the real sizing question is elsewhere. It is not “does the model fit?”, it is: what context length and what level of concurrency do I have to absorb, and how much VRAM is left to house them once the weights are down?
Sources and method
The 2 bytes per parameter in 16-bit rule is a fact: it follows from the IEEE 754 half-precision (FP16) and bfloat16 formats, both encoded on 16 bits. The overhead of 4-bit formats (~4.8 effective bits per parameter for Q4_K_M, or ~42.5 GB for a 70B) is measured on the GGUF k-quant schemes: see the llama.cpp quantization documentation and the GGUF file sizes published on Hugging Face. The Table 1 sizes are orders of magnitude computed from the rule; exact values vary by a few percent from one model to another (shared heads, tied embeddings). The Table 2 tiers rest on the published VRAM capacities of the cards cited (RTX 4060/4070/4090/5090, RTX 3090, A6000, A100/H100); “comfortable” is an estimate incorporating a usual margin for the KV cache and overhead, not a guarantee: the real context and batch decide. The sizing of the KV cache and overhead is detailed in a dedicated article. A French version of this article, published June 1, 2026, is the original.
Image credit. Header photo: NVIDIA GeForce GTX 780 PCB by GBPublic_PR, CC BY 2.0, via Wikimedia Commons, cropped to the die and its ring of memory chips.
