Deep dives
Every LeCompute article available in English: silicon, inference runtimes, edge AI, costs and system observability.
47 articles in English , page 3 of 4
Costs
MLPerf Endpoints measures an inference service through its API at several load levels. Version 0.7 finally traces the curve between throughput, TTFT, TPOT and quality, but it does not normalize cost yet.
Runtimes
MLPerf E2E-RAG times ingestion, retrieval, reranking and four LLM roles inside a multi-hop loop. Its throughput is comparable, but the Offline scenario and 35% reference accuracy bound the verdict.
Silicon
UALink 2.0 standardizes compute inside the network; NVLink 6 integrates it into its switches. Here is what the mechanism changes, and why Helios and Vera Rubin still cannot be ranked.
Runtimes
Expert parallelism moves tokens to weights. DWDP reverses the path, prefetching remote expert weights and removing cross-rank synchronization. The bet works inside a narrow regime.
Silicon
Assess a 96 GB M3 Ultra for local LLMs: 819 GB/s bandwidth, model weights, context and MLX. Includes the distinction from the current M5 Ultra catalog.
Silicon
An RTX PRO 6000 and three RTX 5090s each add up to 96 GB of GDDR7. For an LLM, that is where the equality ends: one pool, one sharded model and three independent replicas pay different costs.
Runtimes
vLLM 0.27 lets every engine serve CPU-resident KV blocks to another peer over NIXL, filters tiers per request and emits routable cache events. The mechanism is shipped; the production speedup is not proven.
Silicon
RTX Spark brings up to 128 GB of unified memory and native CUDA to Windows on Arm. As of August 12, CUDA remains a developer preview, TensorRT-RTX is ready, and the LLM serving stack still needs qualification.
Runtimes
Kimi K3 activates 104 billion parameters out of 2.8 trillion, mixes KDA with MLA across a one-million-token context, and needs an eight-B300 or eight-MI355X node in vLLM's launch recipes.
Runtimes
An inference engine's scheduler does not trade throughput against latency. It trades one latency against another: time to first token versus time between the tokens that follow. Three major engines, three opposite defaults, and nothing tells the operator.
Runtimes
A mixture-of-experts does not remove the cost, it moves it twice: from compute to memory, then from memory to the network. The second shift has a threshold, it can be computed, and it decides your topology.
Costs
A driving model is a function of its data distribution. But the cost of moving it is not where intuition puts it: perception recalibrates for almost nothing, behaviour does not, and validation dominates everything. The entry price for a market is a simulation bill.
