Runtimes & Inference
vLLM, llama.cpp, TensorRT-LLM, KV cache and quantization: how to serve language models efficiently, from the datacenter to a local machine.
4 articles
Runtimes
GGUF is a file format, GPTQ and AWQ are algorithms. At 4 bits, each attacks the outlier problem differently. Anatomy of three constructions, from GPTQ's Hessian to GGUF's k-quants, and of what the choice of format decides for you.
Runtimes
On July 8, 2026, OpenAI disavowed SWE-Bench Pro: ~30% broken tasks. It is the third reference instrument declared dead in three generations. Anatomy of a metrology crisis: noise ceiling, contamination, and a harness effect worth a whole model generation.
Runtimes
NVIDIA Dynamo 1.0, SGLang 0.5.12, TensorRT-LLM 1.3, Tenstorrent Blackhole: prefill/decode disaggregation reaches production. Implementations, benchmarks, and counter-arguments.
Runtimes
The KV cache is no longer a VRAM byproduct you just absorb: it has its own quantization (TurboQuant, ~3 bits), its transfer protocol, its tiered storage (KVBM G1→G4) and its own routing. Anatomy of that shift.
