Why a longer conversation changes memory use
You load a model and a short request works. Add a long document or several users and the server starts rejecting requests. The weight file has not changed, but the state retained for active sequences has grown.
At each attention layer, a token produces key and value vectors. The KV cache retains those projections so that the next generation step can reuse them. Attention still has to consult the relevant history; caching avoids repeated projection work rather than making the history free to read.
Weights have a footprint set by the model and its precision. Cache use also depends on the workload. That distinction explains why a VRAM estimate from parameter count is a starting point rather than a server capacity estimate.
Calculate bytes from the architecture
For dense attention with the same configuration at every layer:
Grouped-query attention shares key/value heads between query heads. Read the KV head count in the configuration rather than substituting the total attention head count. Sliding-window attention, hybrid architectures and compressed latent caches need their own accounting.
Consider an explicit sizing example with 80 layers, eight KV heads and a head dimension of 128. FP16 or BF16 uses two bytes per value, giving 327,680 bytes per retained token. These are chosen architectural inputs, not a measurement or a universal specification for every 70B model.
| Tokens per sequence | One sequence, FP16 | One sequence, FP8 | Four sequences, FP16 |
|---|---|---|---|
| 2,048 | 0.625 GiB | 0.3125 GiB | 2.5 GiB |
| 8,192 | 2.5 GiB | 1.25 GiB | 10 GiB |
| 32,768 | 10 GiB | 5 GiB | 40 GiB |
| 131,072 | 40 GiB | 20 GiB | 160 GiB |
At 32,768 tokens, increasing concurrency from one to four sequences raises FP16 cache use from 10 to 40 GiB. If only 20 GiB remains after the weights and other allocations, the raw budget fits two such sequences before a safety margin. Count retained generated tokens as well as the input prompt.
Paging, quantization and reuse solve different problems
PagedAttention allocates cache in blocks, with a table mapping each sequence to physical storage. A short request need not reserve one contiguous allocation for its maximum possible context. This reduces wasted reservations and fragmentation; it does not compress the key/value vectors, and partially filled blocks can still waste space.
FP8 uses one byte per value instead of two, halving the raw value storage compared with FP16. Total server memory does not halve because weights and other allocations may stay unchanged. Check the supported GPU and attention backend, then validate answer quality. vLLM documents quantization scales and recommends dataset calibration; default scales of one do not establish that accuracy is unchanged.
Prefix caching reuses an already computed shared prefix. Repeated questions about the same document can skip that document’s prefill work. Different documents or evicted prefixes may provide little reuse. The optimization does not eliminate decoding the new answer. Measure cache hits and time to first token on shared-prefix and non-shared workloads.
Turn the estimate into a serving test
Start with memory actually occupied after loading the weights. Add the cache budget for the context lengths and concurrency you intend to support, plus room for buffers and temporary allocations. Test realistic length distributions rather than treating a configured maximum context as the typical request.
Next, measure latency and errors as concurrent traffic increases. The vLLM, llama.cpp and TensorRT-LLM comparison explains how to compare serving implementations. If cache must move between stages or machines, KV cache as the central serving object extends this memory budget into an infrastructure question.
Sources and method
English adaptation published November 3, 2026. Table values are arithmetic from the stated example, not hardware benchmark results. Allocation mechanism: Kwon et al., PagedAttention, SOSP 2023 and vLLM’s PagedAttention design. Implementation details and limits: vLLM quantized KV cache and automatic prefix caching, reviewed on that date.
