What changed since PagedAttention
The KV cache, the store of attention keys and values a model keeps so it never recomputes past tokens, used to be framed as a memory-consumption problem, and our French primer treated it that way: the cache grows linearly with context length and batch size, and it is the cache, not the weights, that saturates VRAM first. PagedAttention An algorithm introduced by vLLM that manages the KV cache like an operating system's virtual memory: in pages, without requiring a contiguous block per request. It eliminates internal and external fragmentation, letting a server handle 2 to 4× more concurrent requests on the same VRAM. , the vLLM scheme that stores that cache in non-contiguous pages the way an operating system handles virtual memory, solved fragmentation; FP8 An 8-bit floating-point format. A versatile working format for inference (and training) on recent GPUs. It halves the memory footprint and bandwidth needed versus FP16, for a marginal accuracy loss on most models. quantization (one byte per value instead of two) doubled capacity; prefix caching removed redundant recomputation. Those three levers remain the foundation. But each treats the KV cache as a local byproduct: a tensor the GPU owns, manages, and shares with no one.
What happened between late 2025 and May 2026 goes beyond that frame. The KV cache has acquired its own specialized quantization, distinct from the weights’. It now has a transfer protocol for migrating from one node to another. It lives in a four-tier memory hierarchy, from GPU HBM High Bandwidth Memory. Memory stacked in layers and soldered right next to the GPU, with several TB/s of bandwidth (versus ~50 GB/s for DDR5). Indispensable beyond a certain model size. (the high-bandwidth memory soldered next to the GPU) down to S3 object storage. And the inference scheduler makes its routing decisions based on the state of the cache on each node.
The KV cache is no longer a side effect. It is the central object around which the serving stack is reorganizing.
TurboQuant: the cache drops to ~3 bits
FP8 quantization of the KV cache (one byte per value instead of two) had been a production default since 2024. Going lower without hurting quality looked out of reach: the KV cache changes at every token, unlike the weights, which you calibrate once. Aggressive quantization schemes (4-bit, 2-bit) produced measurable artifacts on generation benchmarks.
TurboQuant, published at ICLR 2026 by Google Research and NYU (arXiv 2504.19874), solves this with two chained mechanisms.
The first, PolarQuant, applies a random orthogonal rotation to the K and V vectors before quantization. The idea is not to approximate the values as they are: it is to transform them into a space where their distribution becomes near-Gaussian, hence better suited to uniform quantization. The rotation is invertible and cheap: a randomly drawn orthogonal matrix, applied by multiplication, and undone after the attention computation. What would be expensive to store (the matrix) is handled by a fixed seed: the same matrix is rebuilt on every call, never kept.
The second, QJL (Quantized Johnson-Lindenstrauss), corrects the quantization residual with a 1-bit random projection. Instead of storing the error at full resolution, QJL projects it into a subspace and keeps only the sign of each projected component, one bit. The Johnson-Lindenstrauss lemma guarantees that the distance between vectors is preserved with high probability in the projected subspace, which is enough for the attention computation. The combined result: about 3 bits per coordinate for a quality indistinguishable from the baseline, 2.5 bits with marginal degradation.
The vLLM 0.20 integration has been live since April 15, 2026 (PR #38479). The CLI flag is --kv-cache-dtype turboquant_3bit_nc. A FlashAttention 3/4 extension for prefill The opening phase of LLM inference: every token of the prompt is processed at once. High arithmetic intensity, so the GPU saturates its Tensor Cores. The opposite of the decode phase that follows. followed in PR #40092. The vLLM blog of May 11, 2026 published the first systematic study. This is no longer a research paper, it is a configuration parameter.
RocketKV: smart eviction
TurboQuant compresses every value in the cache. RocketKV (arXiv 2502.14051, NVlabs, ICML 2025) takes the problem from the other end: not storing the entries attention will never look at.
The mechanism runs in two stages. The first, SnapKV++, performs a coarse eviction: at regular intervals it evaluates which cache blocks contributed least to the recent attention score and drops them. It is a high-pass filter on temporal relevance: old, never-consulted blocks disappear. The second stage applies a hybrid top-k sparse attention over the surviving cache: instead of computing attention over every remaining vector, it keeps only the k most relevant entries, selected by a lightweight score.
Measured on H100: up to 3× decode The autoregressive generation phase of an LLM: one token is produced at a time, re-reading the whole KV cache. Arithmetic intensity is very low, so the GPU spends most of its time waiting on memory. A real inference service is almost always dominated by decode. speedup, 31% peak-memory reduction. Its integration into TensorRT-LLM as a sparse-attention backend makes it complementary to quantization: TurboQuant shrinks the size of each entry, RocketKV shrinks the number of entries.
FlashAttention 4: prefill comes down a step
The attention computation itself crossed a threshold in March 2026. FlashAttention 4 (arXiv 2603.05451, Zadouri, Dao et al., Together AI) reaches ~1,613 TFLOPS/s at 71% utilization on Blackwell Tensor Cores Hardware units specialized in low-precision matrix multiplication, introduced by NVIDIA with Volta (2017). Each generation adds supported formats: FP16 → FP8 (Hopper) → FP6/FP4 (Blackwell). They run the bulk of the inference compute. (SM100+): the paper measures ~1.3× cuDNN and ~2.7× Triton on B200. The gain does not come from an “inference-only” mode: FA4 does have a backward pass and serves training too. It comes from an algorithm/kernel co-design: asymmetric pipelining cut for Blackwell, a faster exponential, better overlap of the MMAs (the Tensor Core matrix-multiply-accumulate operations).
vLLM 0.20 uses it as the default MLA backend for prefill on Blackwell (SM100+); on Hopper, FA3 fills that role. The link to the KV cache is direct: the faster the prefill, the less the cache-filling time weighs in the TTFT (time to first token), which makes prefix caching less critical on short sequences and even more valuable on long ones.
The KV cache memory hierarchy: KVBM G1→G4
Until 2025, the KV cache lived in HBM or did not exist. CPU offloading existed as a prototype (vLLM, LMCache), but with no tier management, no migration policy, no persistent storage. The cache was an ephemeral object, destroyed when the request ended.
NVIDIA Dynamo 1.0, generally available since March 16, 2026, changes that posture radically. Its KVBM (KV Block Manager) subsystem organizes the cache into a four-tier storage hierarchy that borrows its logic from CPU memory hierarchies (L1 / L2 / L3 / DRAM), but at datacenter scale.
G1: GPU HBM. The hot tier, where the cache lives during the attention computation. Access latency on the order of a microsecond, bandwidth of several TB/s. This is the tier PagedAttention has managed since 2023. The difference is that KVBM treats it as one floor among four rather than the cache’s only living space.
G2: CPU RAM, potentially distributed. The cache of a finished or interrupted request migrates to host RAM instead of being destroyed. If a later request presents the same prefix, the cache is promoted back to G1 with no recomputation. The cost: one PCIe round trip, on the order of a millisecond for a typical block. Dynamo distributes G2 across several CPU nodes: a prefix’s cache is not confined to local RAM, it is reachable from any node in the cluster.
G3 / G3.5: local SSDs and ICMS. For caches whose reuse frequency no longer justifies RAM, KVBM steps down to flash storage. The standard G3 tier uses local or pooled NVMe. The G3.5 tier is specific to the Vera Rubin platform: ICMS (Inference Context Memory Storage), flash storage integrated into the BlueField-4 DPUs and driven by the DOCA Memos framework. ICMS is sized for the KV cache, not for generic storage, and offers a data path that bypasses the host CPU through the NIC.
G4: remote object storage. The archive tier: S3, Azure Blob Storage. A long-session cache (an agent’s context over several hours, the history of a RAG workflow) persists in G4 and reloads on demand. Access latency is measured in tens of milliseconds, but recomputing the prefix would have cost seconds. G4 offloading is part of the March 2026 Dynamo 1.0 release, not a feature bolted on afterward.
The migration policy between tiers is governed by the blocks’ reuse frequency. A block referenced once drops quickly to G3/G4; a block several requests share (system prompt, anchored RAG document) stays in G1 or G2. It is exactly the logic of a CPU cache with an extended LRU policy, applied to blocks of tens of megabytes instead of 64-byte lines.
SGLang: HiSparse and HiCache
SGLang 0.5.12 attacks the same hierarchy with two distinct systems, each targeting a segment of the problem.
HiSparse handles GPU → CPU offloading for models that support sparse attention, so far DeepSeek-V3.2 and GLM-5.1, which use the DSA (DeepSeek Sparse Attention) architecture. The mechanism is proactive: HiSparse identifies inactive KV cache entries before they saturate HBM and moves them to host memory, while keeping a hot buffer in HBM for high-access-frequency entries. Measured effect: over 3× the baseline throughput at 256 concurrent requests, up to 5× on long contexts. The first public description came in the LMSYS blog of April 10, 2026.
HiCache is a separate, broader system. It implements a UnifiedRadixTree with Sliding Window Attention support, which lets it manage fixed-attention-window models (DeepSeek V4 included) in a shared cache tree. SSD offloading goes through the Mooncake store, a distributed cache-transfer engine that also shows up in TensorRT-LLM. HiCache and HiSparse are complementary, not alternatives. HiSparse handles GPU↔CPU migration, HiCache handles the cache hierarchy extended down to SSD.
Transfer connectors: the KV cache in transit
In a disaggregated serving pipeline, where one node runs prefill and another runs decode, the KV cache has to travel. A 70-billion-parameter model with 8k tokens of context produces ~2.6 GB of KV cache in FP16; in FP4 A 4-bit floating-point format, the 2026 frontier of high-throughput inference. Four times less memory than FP16, but a very narrow dynamic range: it only holds up with fine-grained scaling through block formats (MXFP4, NVFP4). (four bits per value), that is still hundreds of megabytes to move between nodes before decode can begin. Without an optimized protocol, this transfer cancels the gain of disaggregation Splitting the inference pipeline into distinct phases (prefill, decode) run on specialized node pools. It lets you assign compute-bound hardware to prefill and memory-bound hardware to decode, instead of running both on the same GPU that stays underused on one of the two phases. .
Three connectors emerged in parallel.
KV Cache Connector API (TensorRT-LLM). Introduced in TRT-LLM v1.1 (not v1.3, which refines the implementation), this API standardizes the cache’s serialization, transfer and restore operations. v1.3 rc12 adds shortcuts for lmcache and kvbm; rc15 introduces a cache_salt_id to identify cache blocks by signature rather than by address. The KV cache becomes an addressable object, not an anonymous buffer.
NIXL (vLLM). vLLM’s internal transfer connector, redesigned in version 0.21 (May 2026) to support bidirectional transfers between P (prefill) and D (decode) nodes. v0.21 also adds the MooncakeStoreConnector for distributed offloading, and DCP/PCP support in the offloading connector.
LMCache. An independent project that combines CacheGen (KV cache compression), CPU offloading, and distributed caching. Compatible with vLLM. The approach is complementary: LMCache compresses before the transfer, which cuts the inter-node bandwidth needed.
CacheBlend (EuroSys 2025) rounds out the picture with a selective reuse mechanism: instead of recomputing a RAG document’s KV cache on every request, CacheBlend restores the pre-existing parts and recomputes only the new tokens. Measured result: 2.2 to 3.3× TTFT reduction on RAG workloads.
KV-aware routing: the scheduler changes its criterion
All these storage and transfer mechanisms would be useless if the scheduler kept routing requests without regard for the state of the cache. This is the last piece of the puzzle, and the one that produces the most spectacular gains.
Dynamo 1.0’s Kubernetes Inference Gateway plugin embeds a token-aware routing algorithm. Instead of assigning a request to the least-loaded node (weighted round-robin, the norm), the router first checks which nodes already hold the KV cache for the incoming request’s prefix. If a node in G1 or G2 holds the cache for the system prompt or the RAG document, the request goes straight there: the prefix’s prefill is short-circuited, and the TTFT drops.
NVIDIA measures a 20× TTFT reduction and a 4× end-to-end latency reduction on workloads with recurring prefixes. Baseten, in a Dynamo deployment on Qwen3 Coder 480B with ~50k-token prompts, reports a 50% TTFT reduction, a 61% gain in requests per second, and a 62% gain in tokens per second. Their blog headline says “2x faster inference”: it is specifically the TTFT that is halved, not overall throughput.
Comparison: the KV cache by runtime
| Feature | vLLM 0.20-0.21 | SGLang 0.5.12 | TRT-LLM v1.1-1.3 | Dynamo 1.0 |
|---|---|---|---|---|
| FP8 KV quantization | ✓ | ✓ | ✓ | ✓ (via runtime) |
| TurboQuant ~3 bits | ✓ (v0.20, PR #38479) | — | — | — |
| Sparse attention (RocketKV) | — | HiSparse (DSA) | ✓ (sparse backend) | — |
| FlashAttention 4 (prefill) | ✓ (SM90+, default) | — | — | — |
| Smart CPU offloading | ✓ (reuse frequency) | ✓ (HiSparse) | ✓ (host offloading) | G2 (KVBM) |
| SSD offloading | — | ✓ (HiCache + Mooncake) | — | G3 / G3.5 (KVBM) |
| Object-storage offloading | — | — | — | G4 (S3 / Azure Blob) |
| P/D transfer (disaggregation) | ✓ (NIXL, bidirectional) | — | ✓ (KV Cache Connector API) | ✓ (KVBM + NIXL) |
| KV-aware routing | — | — | — | ✓ (K8s Inference Gateway) |
| FlexKV (variable length) | ✓ (v0.17.2+, Tencent TACO) | — | — | — |
| Automatic prefix caching | ✓ | ✓ | ✓ | ✓ |
If you are sizing your serving stack, three readings stand out. vLLM concentrates the quantization innovations (TurboQuant, FlexKV, FA4): it is the runtime where the cache is most compressed per entry. SGLang concentrates the local hierarchy innovations (HiSparse + HiCache): it is the runtime that pushes the cache furthest off the GPU, but for a subset of models. Dynamo brings coordination at the cluster level (KVBM + KV-aware routing) and plugs into vLLM or TRT-LLM as the underlying runtime. TRT-LLM provides the standardized connector API and the RocketKV backend, but delegates orchestration to Dynamo.
vLLM 0.21 and the full P/D pipeline
vLLM version 0.21 (May 15, 2026) marks the moment the KV cache becomes a first-class object in the disaggregated pipeline. The HMA (Heterogeneous Memory Architecture) integration with the offloading subsystem unifies the memory paths: a KV block follows a G1 → CPU → back-to-G1 trajectory with no format change. Bidirectional transfers between P and D nodes go through the redesigned NIXL connector: a decode node can now send cache back to a prefill node, not just receive it.
The MooncakeStoreConnector opens distributed offloading: a P node’s cache can transit through the Mooncake store to a remote D node without going through the P node’s host RAM. And DCP/PCP support in the offloading connector lays the ground for architectures where the prefill and decode nodes are not on the same physical network.
The TOKENSPEED_MLA attention backend, optimized for Blackwell GPUs, rounds out the picture by accelerating the MLA computation (Multi-Latent Attention, DeepSeek V3/V4’s attention architecture) on the newest hardware.
Multimodal extends the hierarchy
Dynamo 1.0 is not limited to text. Its multimodal E/P/D (Embedding / Prefill / Decode) pipeline adds a third cache tier: the embedding cache for precomputed visual representations. On Qwen3-VL-30B-A3B-Instruct in FP8 on GB200, NVIDIA measures a 30% TTFT reduction and 25% higher throughput on image requests, a gain that comes entirely from the fact that the visual embedding, expensive to compute, is computed only once.
Video follows the same logic. FastVideo + SGLang Diffusion generates a 5-second 1080p video in ~4.5 s on a single B200, a throughput that makes real time plausible for the first time on standard datacenter hardware.
What comes next: BlueField-4 and the cache as network infrastructure
The Vera Rubin platform writes the KV cache into network silicon. The BlueField-4 DPU, announced at CES in January 2026 and expected in the second half of 2026, embeds DOCA Memos, a framework for KV cache communication and storage directly in the DPU. The G3.5 tier (ICMS) is not an SSD the DPU addresses: it is flash storage built into the network card, reachable without crossing the host CPU or the main PCIe bus.
There the KV cache is treated as a network object, not as application data. The DPU can route it, replicate it, compress it on the data path, the same way a smart network switch manipulates packets without the CPU getting involved. This is the next frontier: the cache is no longer only managed by the runtime, it is transported by the network infrastructure itself.
Silicon is reorganizing around the cache too, and not only at NVIDIA: Google’s TPU 8i tripled its on-chip SRAM (384 MB) precisely to house a larger KV cache closer to the compute.
MLPerf v6.0: the cache in the official results
The MLPerf Inference v6.0 round (April 1, 2026) brings an indirect but significant confirmation. NVIDIA’s submissions on B200 use the FP8 KV cache as the reference configuration, not FP16, not FP32. It is the first time a reduced cache format appears in official benchmark submissions as a baseline, not as an optional optimization. AMD, for its part, uses a pre-shuffled KV layout, a pre-optimized memory arrangement of the cache to reduce HBM bank conflicts. Even among the challengers, the cache has become a first-order optimization axis.
Conclusion
The KV cache’s trajectory in 2026 follows the one CPU cache memory traced in the 1990s: a performance byproduct turned full-fledged subsystem, with its own hierarchy, its own policies, and its own communication primitives. The next steps are visible. AMD’s pre-shuffled KV layout and LMCache’s CacheGen compression point toward a convergence on standardized cache formats, a GGUF of the KV cache portable across runtimes. DOCA Memos and BlueField-4 write the cache into the DPU, which moves the management boundary from the runtime toward the network. And Dynamo’s KV-aware routing raises the next question: if the scheduler knows the state of every node’s cache, is the next step not a market for the cache, where a node that has invested compute in a prefix sells it to the nodes that need it, instead of letting it die?
Sources and method
TurboQuant. Zandieh et al. (Google Research), ICLR 2026, arXiv 2504.19874. vLLM integration: PR #38479 (April 15, 2026), PR #40092 (FA3/FA4 extension). vLLM blog “First Comprehensive Study of TurboQuant” (May 11, 2026). The 6× memory reduction and 8× attention speedup are the paper’s claims, FP32 baseline; verified fact (paper + vLLM reproduction).
RocketKV. Zhang et al., ICML 2025, arXiv 2502.14051. 3× decode speedup and 31% peak-memory reduction on H100; verified fact (paper). TRT-LLM integration as sparse backend; verified fact (TRT-LLM release notes).
FlashAttention 4. Zadouri, Dao et al., arXiv 2603.05451, March 2026 (“Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling”). ~1,613 TFLOPS/s, 71% utilization on B200; ~1.3× cuDNN, ~2.7× Triton; verified fact (paper). FA4 targets Blackwell (SM100+) and has a backward pass (training). MLA prefill backend of vLLM 0.20 on Blackwell; verified fact (vLLM 0.20 release notes).
vLLM 0.20 / 0.21. Release notes: vLLM 0.20.0 (April 27, 2026), vLLM 0.21.0 (May 15, 2026). FlexKV (PR #34328, Tencent TACO) shipped since v0.17.2, not introduced in v0.20; verified fact. Smart CPU offloading (PR #35342) appeared in v0.18.0; verified fact.
SGLang 0.5.12: HiSparse and HiCache. LMSYS blog, April 10, 2026: HiSparse. 3-5× throughput at 256 concurrent requests (DSA workloads); verified fact (LMSYS blog). HiSparse and HiCache are distinct systems, not interchangeable; verified fact (SGLang 0.5.12 release notes).
NVIDIA Dynamo 1.0: KVBM. NVIDIA Developer blog: Dynamo 1.0 Production Ready (March 16, 2026). G1-G4 tiers, S3/Azure Blob offloading; verified fact (official blog). Token-aware routing: 20× TTFT, 4× e2e latency; NVIDIA figures on recurring-prefix workloads, to be read as an “up to” bound (not independently reproduced).
Baseten. Blog: How Baseten Achieved 2x Faster Inference with NVIDIA Dynamo. 50% TTFT reduction, 61% RPS, 62% TPS on Qwen3 Coder 480B (~50k-token input). The “2×” is specifically the TTFT; verified fact (Baseten blog, May 2026).
TensorRT-LLM: KV Cache Connector API. Introduced in TRT-LLM v1.1, not v1.3; verified fact (v1.1 release notes). v1.3 rc12: lmcache/kvbm shortcuts; rc15: cache_salt_id; verified fact (GitHub changelogs).
BlueField-4 and DOCA Memos. CES January 2026 announcement. ICMS (Inference Context Memory Storage) and DOCA Memos; verified fact (NVIDIA CES 2026 press release). H2 2026 availability; verified fact (official roadmap).
CacheBlend. EuroSys 2025. 2.2-3.3× TTFT reduction on RAG workloads; verified fact (EuroSys paper).
MLPerf v6.0. April 1, 2026. FP8 KV cache in NVIDIA B200 submissions, AMD pre-shuffled KV layout; verified fact (public MLPerf results).
Multimodal Dynamo. +30% TTFT, +25% throughput on Qwen3-VL-30B/GB200 (image requests); verified fact (NVIDIA Dynamo blog). FastVideo 1080p 5s in ~4.5s on a B200; verified fact (NVIDIA demo).
This article is the English adaptation of a French original, published on May 20, 2026.
