Runtimes
NVIDIA PAIR: when more machines make local AI faster
Your GPU is busy while another computer sits idle. NVIDIA PAIR can send it requests, but whether you finish sooner depends on how the work is shared.
Runtimes & Inference
Runtimes: serving LLMs efficiently
vLLM, llama.cpp, TensorRT-LLM, KV cache and quantization: how to serve language models efficiently, from the datacenter to a local machine.
11 articles
Articles: Runtimes
Runtimes
MLPerf RAG finally measures the full pipeline, but not yet the service
MLPerf E2E-RAG times ingestion, retrieval, reranking and four LLM roles inside a multi-hop loop. Its throughput is comparable, but the Offline scenario and 35% reference accuracy bound the verdict.
Runtimes
DWDP for MoE inference: move weights, not tokens
Expert parallelism moves tokens to weights. DWDP reverses the path, prefetching remote expert weights and removing cross-rank synchronization. The bet works inside a narrow regime.
Runtimes
vLLM 0.27 turns the KV cache into peer-to-peer memory
vLLM 0.27 lets every engine serve CPU-resident KV blocks to another peer over NIXL, filters tiers per request and emits routable cache events. The mechanism is shipped; the production speedup is not proven.
Runtimes
Kimi K3 opens 2.8T parameters, not a local model
Kimi K3 activates 104 billion parameters out of 2.8 trillion, mixes KDA with MLA across a one-million-token context, and needs an eight-B300 or eight-MI355X node in vLLM's launch recipes.
Runtimes
Latency against latency: what your scheduler actually decides
An inference engine's scheduler does not trade throughput against latency. It trades one latency against another: time to first token versus time between the tokens that follow. Three major engines, three opposite defaults, and nothing tells the operator.
Runtimes
MoE serving: at what batch size does the network take over?
A mixture-of-experts does not remove the cost, it moves it twice: from compute to memory, then from memory to the network. The second shift has a threshold, it can be computed, and it decides your topology.
Runtimes
GGUF vs GPTQ vs AWQ: a format, two algorithms, one problem
GGUF is a file format, GPTQ and AWQ are algorithms. At 4 bits, each attacks the outlier problem differently. Anatomy of three constructions, from GPTQ's Hessian to GGUF's k-quants, and of what the choice of format decides for you.
Runtimes
Nobody knows how to measure coding agents anymore
On July 8, 2026, OpenAI disavowed SWE-Bench Pro: ~30% broken tasks. It is the third reference instrument declared dead in three generations. Anatomy of a metrology crisis: noise ceiling, contamination, and a harness effect worth a whole model generation.
Runtimes
Prefill/decode disaggregation reaches production: state of play, May 2026
NVIDIA Dynamo 1.0, SGLang 0.5.12, TensorRT-LLM 1.3, Tenstorrent Blackhole: prefill/decode disaggregation reaches production. Implementations, benchmarks, and counter-arguments.
Runtimes
The KV cache is no longer a side effect: it is the center of LLM serving in 2026
The KV cache is no longer a VRAM byproduct you just absorb: it has its own quantization (TurboQuant, ~3 bits), its transfer protocol, its tiered storage (KVBM G1→G4) and its own routing. Anatomy of that shift.