The next token now depends on where the past lives
During prefill The opening phase of LLM inference: prompt tokens are processed in parallel to build the context state. That reuse raises arithmetic intensity and can make the phase compute-bound; the exact regime depends on the model, batch, context and backend. , an LLM turns prompt tokens into key and value vectors. The KV cache The stored key and value vectors an LLM has already computed for every token it has processed. It avoids recomputing attention over the whole history, at the cost of a memory footprint that grows with the context length. retains those vectors so decode The autoregressive generation phase of an LLM: one token is produced at a time while rereading weights and the relevant context state. At low batch sizes, that traffic can make decode memory-bandwidth-bound. Its share of total cost still depends on input and output lengths, batching and cache reuse. does not reprocess the complete prompt before generating each new token. vLLM had already paged this memory, shared matching pages between local requests and offloaded completed blocks into CPU RAM.
Reuse still stopped at a process boundary. If the same prefix reached another worker, that GPU recomputed the layers or relied on a prefill/decode setup with predetermined producer and consumer roles. A router could select a warm worker, but a cold worker could not query any neighbor as a cache source through one generic tier.
vLLM 0.27 adds that move. Each instance can expose a P2P tier and answer lookups against its CPU cache. For each request, the orchestrator names a remote peer. The consumer sends the block hashes it needs; the producer returns hit, miss or pending status; NIXL moves the matched pages.
This turns scheduling into a data-locality decision. The controller no longer picks only the GPU that will execute a request. It compares the cost of moving the request’s past with the cost of rebuilding it.
Three levels, with CPU memory as the gateway
vLLM’s offloading connector separates compute memory from retention tiers. The GPU executes attention. Pinned CPU RAM is the primary tier and the only level with direct GPU access. Secondary tiers can extend capacity into a filesystem, object store or remote peer.
This is not a universal direct GPU-to-GPU read. A computed block first lands in the producer’s host memory through DMA. The remote peer pulls it through a NIXL backend, stores it in its own CPU region and promotes it to GPU. The guide defaults to UCX and allows backends such as Mooncake, GDS_MT and libfabric.
Control uses a separate ZMQ path. Sessions carry lookups, replies and completion state. Every data-parallel rank listens on a port derived from one base. Cross-host deployments must replace the default localhost binding with a routable address; vLLM does not discover that callback address automatically.
Separating control from tensor movement keeps large data out of the coordinator. It also creates two failure planes. A session may reach a peer while its NIXL backend is unhealthy, or data transport may be ready while the ZMQ port is blocked.
The peer never decides where to search
The P2P tier accepts a source address; it does not maintain the global cache map. That belongs to a router, Endpoint Picker or external scheduler. The orchestrator must know which peer holds the prefix, its data-parallel rank, control port and the transaction ID shared across the transfer.
| Key | Receiving instance | Effect | Required fields |
|---|---|---|---|
| remote_decoder | Prefill producer | Keeps blocks for a decoder | kv_request_id |
| remote_prefiller | Decode consumer | Pulls cache from one prefiller | id, host, port |
| remote_kv_source | P2P consumer | Looks up blocks currently held by a peer | id, host, port |
| No key | Local peer | Uses CPU tier without remote exchange | No remote address |
| decoder + kv_source | Producer and consumer | Fetches a prefix, then serves the continuation | Allowed combination |
The remote_decoder plus remote_kv_source combination captures the new topology. A prefill worker can fetch an existing prefix from one peer, compute only the extension and retain new blocks for a different decoder. Roles form a request-specific chain instead of dividing processes into permanent prefiller and decoder classes.
Contradictory combinations are rejected. A consumer cannot pull simultaneously from remote_prefiller and remote_kv_source because two sources would compete over one block sequence. The orchestrator has to produce a coherent path before vLLM transfers data.
Lookup, promotion and compute form a three-way decision
The consumer sends block hashes. The source returns HIT when data is available, MISS when it is absent or HIT_PENDING when an announced write has not completed. Pending waits are bounded so a stalled producer cannot defer a request forever.
On a hit, data enters the target CPU region and becomes eligible for GPU promotion. On a miss, the engine computes the prefix locally. That fallback is part of correctness: a distributed map always trails evictions, request completion and failures by some amount.
Version 0.27 also adds per-request tier filtering. kv_load_tiers can restrict secondary tiers by medium and locality. A latency-sensitive request may accept local CPU memory and reject storage; an asynchronous job may search larger tiers. The primary CPU tier remains in every path because it is the promotion target before GPU memory.
The control plane now applies cost policy across local HBM, local CPU, a remote peer, storage and recomputation. The first available hit is not necessarily the cheapest one when its source is saturated or distant.
Self-describing events give the router a map
A KV-aware router needs to link a content-derived block identity with the worker that owns it. A local block ID alone cannot do that. The route planner needs tokens, parent hash, block size and locality to derive the same chain as the engines.
vLLM 0.27 enables self-describing events with TieringOffloadingSpec. Normal GPU-to-CPU stores use the existing path. A promotion from a secondary tier records metadata when lookup reaches a hit; the resulting CPU BlockStored event can then carry a complete payload. NVIDIA Dynamo can index the promoted copy and route later requests to that worker.
The implementation documents important limits. A promotion event remains incomplete if no local request observes the hit before event translation. Sliding-window and SSM groups still carry placeholder payloads. No in-tree secondary tier emits removal events, and reset may leave older queued events visible after state has been cleared.
Eviction becomes an extension point
The CPU cache already included LRU and ARC. Version 0.27 adds CachePolicyFactory, allowing an external package to register a policy by name without replacing private manager state. Built-in LRU and ARC behavior stays unchanged; construction becomes extensible like offloading specs and connectors.
This matters in a hierarchy. Local LRU ignores recompute cost, global popularity, copies on other peers and network volume. An external policy can include those signals or protect shared system prompts while expiring conversation-specific branches.
Selective offload complements eviction. Each request can cap how many of its tokens become eligible for offload. Persisting a shared system prompt while skipping unique late-turn tokens reduces writes that have little chance of reuse. A cache wins by refusing unprofitable data movement, not by filling every available byte.
When does a remote hit beat a new prefill?
A hit avoids model computation over L prefix tokens. Its cost includes control lookup, peer transfer and CPU-to-GPU promotion. The producer also paid an earlier GPU-to-CPU store. A useful decision boundary is:
lookup time + bytes transferred / effective bandwidth + GPU promotion < avoided prefill time.
Length alone does not decide. A large dense model spends more compute per prefill token than a small model, making the same transfer easier to justify. A model with a large KV footprint moves more bytes for the same prefix. RDMA-class transport can make remote memory competitive; congested Ethernet can make recomputation faster.
Hit rate matters twice. It amortizes the original store and limits control traffic wasted on misses. Chat traffic with little prefix sharing can gain nothing or regress slightly. Agent traffic that reuses a repository, policy prompt and common history can avoid tens of thousands of prefill tokens.
vLLM’s CPU offloading benchmarks report 2x to 22x lower TTFT depending on prompt length on one H100, and up to 9x throughput across 10,000 requests of 512 tokens. They disable the GPU cache, exclude warm-up from measured time and use a single node with 500 GB of RAM. The results establish that CPU reuse can beat recomputation. They do not benchmark the new P2P hop.
The published 97% validates transfer, not performance
The P2P pull request includes a live two-pod test. The prefiller runs TP1, the decoder TP2, the model is Llama 3.2 1B and the prompt contains 234 tokens. A long continuation and deterministic counting case pass. The decoder reports a 97.0% external prefix-cache hit rate, above its 90% threshold.
That experiment answers one core question: can the protocol recover correct tensors across different parallelism degrees? It can in this case. It does not answer operational questions. A 1B model and 234 tokens produce little KV data, do not saturate the network and do not expose control-plane cost under thousands of concurrent sessions.
No vLLM 0.27 P2P result compares TTFT, inter-token latency, goodput, bytes transferred or CPU utilization against recomputation. The changelog ships a capability rather than a speedup. Reading a 97% hit rate as a 97% latency improvement would confuse blocks found with time saved.
The release remains a moving surface
vLLM 0.27.0 combines 561 commits and moves to PyTorch 2.13, torchvision 0.28 and Triton 3.7.1. The release notes call this a breaking environment change. The same release lands Kimi K3, deeper FlashAttention 4 integration, simplified fault tolerance, hybrid-model prefill/decode transfer and early sm_107 Rubin support.
Version 0.27.1 shipped on August 11, less than fourteen hours after 0.27.0, with a DSpark fix. That patch does not target P2P, but it illustrates branch velocity. Operators should pin the image, PyTorch, Triton, FlashInfer, NIXL and router commit, then rerun correctness before load tests.
The Q3 roadmap still lists “production-ready distributed and multi-tier KV cache offloading” as a goal. It also asks for Mooncake P2P events, session hints and an AMD RDMA parity path. The primitive is merged; full production maturity remains work in progress.
Validation must start from a real trace
A useful test replays production sessions with their prefix distribution, lengths and gaps between turns. It compares GPU-only caching, GPU plus CPU, storage-backed hierarchy and P2P on the same model, GPU count, output quality and concurrency budget.
Metrics must separate hit rate by tier, lookup time, bytes read and written, promotion time, p50/p99 TTFT, SLO-compliant throughput, recomputation and eviction before reuse. The router should expose how many choices pointed at a peer that no longer held the block. Failure tests should cut ZMQ control, NIXL transport and the source after route selection.
Security belongs in the protocol. A prefix hash can reveal that content was observed, and a KV block belongs to one model, weight version, tenant and cache configuration. Key isolation and network ACLs must prevent one tenant’s peer from serving another tenant’s state. vLLM supplies transfer primitives; multi-tenant boundaries remain an architecture responsibility.
KV cache is becoming a distributed system
PagedAttention turned GPU memory into managed pages. CPU offloading added a larger, slower tier. vLLM 0.27 takes the next step: a page can become useful on an engine that never computed it, and workers no longer have permanent roles in its path.
That makes KV cache a distributed data product with an index, locality, eviction policy, events, weak consistency and a control plane. The largest gain will not come from adding the most tiers. It will come from choosing a transfer only when the avoided computation is worth more than the bytes moved. The runtime now exposes the mechanism; routers and measurements have to prove the decision.
Sources and method
- vLLM 0.27.0 release notes, August 10, 2026 and vLLM 0.27.1, August 11, 2026. Primary sources for release scope, dependencies and the immediate patch. The notes claim no P2P speedup.
- vLLM KV Offloading Usage Guide, accessed August 12, 2026. Primary source for the GPU/CPU/secondary hierarchy, backends, network variables, per-request roles, filtering and token caps.
- vLLM PR #48021, “Generic P2P secondary tier,” merged July 26, 2026. Primary source for the ZMQ/NIXL protocol, lookup states and two-pod test. The 97% figure is a validation hit rate on Llama 3.2 1B, not a speed measurement.
- vLLM PR #48123, per-request filtering, PR #48679, self-describing events and PR #49114, CachePolicyFactory. Primary sources for new policies and their documented limitations.
- vLLM, “Inside vLLM’s New KV Offloading Connector,” January 8, 2026. Project source for the DMA path and CPU benchmarks. We do not transfer those gains to P2P: the experiment uses one H100, 500 GB RAM, disabled GPU prefix caching and excludes warm-up.
- vLLM Q3 2026 roadmap. Primary source for open goals around production-ready distributed caching, P2P events and AMD parity.
