Runtimes
LLM KV cache: calculate memory before increasing context or concurrency
A worked KV cache calculation in GiB, with the limits of PagedAttention, FP8 quantization and prefix caching when sizing an inference server.
Runtimes & Inference
Runtimes: serving LLMs efficiently
vLLM, llama.cpp, TensorRT-LLM, KV cache and quantization: how to serve language models efficiently, from the datacenter to a local machine.
16 articles , page 1 of 2
Articles: Runtimes
Runtimes
vLLM vs llama.cpp vs TensorRT-LLM: choose a runtime and test it fairly
Compare model support, memory, continuous batching and deployment paths before measuring useful throughput and latency on your own workload.
Runtimes
DeepSeek opens DeepInfer: model weights are only the beginning
DeepSeek has released its inference engine. The code shows how it speeds up models, and how much infrastructure still separates that engine from a production service.
Runtimes
Jev: why generate text when your application needs a decision?
Pick a category, route a request, score a document. Jev returns answers your code can use directly. Here is what that specialization changes and how to evaluate it.
Runtimes
Speculative decoding: when a small model makes your LLM faster
Your model fits in memory, but the answer arrives slowly. A smaller model can draft the next tokens. Here is when that extra work pays off.
Runtimes
NVIDIA PAIR: when more machines make local AI faster
Your GPU is busy while another computer sits idle. NVIDIA PAIR can send it requests, but whether you finish sooner depends on how the work is shared.
Runtimes
MLPerf RAG finally measures the full pipeline, but not yet the service
MLPerf E2E-RAG times ingestion, retrieval, reranking and four LLM roles inside a multi-hop loop. Its throughput is comparable, but the Offline scenario and 35% reference accuracy bound the verdict.
Runtimes
DWDP for MoE inference: move weights, not tokens
Expert parallelism moves tokens to weights. DWDP reverses the path, prefetching remote expert weights and removing cross-rank synchronization. The bet works inside a narrow regime.
Runtimes
vLLM 0.27 turns the KV cache into peer-to-peer memory
vLLM 0.27 lets every engine serve CPU-resident KV blocks to another peer over NIXL, filters tiers per request and emits routable cache events. The mechanism is shipped; the production speedup is not proven.
Runtimes
Kimi K3 opens 2.8T parameters, not a local model
Kimi K3 activates 104 billion parameters out of 2.8 trillion, mixes KDA with MLA across a one-million-token context, and needs an eight-B300 or eight-MI355X node in vLLM's launch recipes.
Runtimes
Latency against latency: what your scheduler actually decides
An inference engine's scheduler does not trade throughput against latency. It trades one latency against another: time to first token versus time between the tokens that follow. Three major engines, three opposite defaults, and nothing tells the operator.
Runtimes
MoE serving: at what batch size does the network take over?
A mixture-of-experts does not remove the cost, it moves it twice: from compute to memory, then from memory to the network. The second shift has a threshold, it can be computed, and it decides your topology.