Where things stood
On May 17, 2026, our article on the Vera Rubin platform laid out a thesis: LLM inference is no longer a homogeneous GPU problem. prefill The opening phase of LLM inference: every token of the prompt is processed at once. High arithmetic intensity, so the GPU saturates its Tensor Cores. The opposite of the decode phase that follows. , the phase that processes the whole prompt at once, is compute-bound: it saturates the Tensor Cores Hardware units specialized in low-precision matrix multiplication, introduced by NVIDIA with Volta (2017). Each generation adds supported formats: FP16 → FP8 (Hopper) → FP6/FP4 (Blackwell). They run the bulk of the inference compute. . decode The autoregressive generation phase of an LLM: one token is produced at a time, re-reading the whole KV cache. Arithmetic intensity is very low, so the GPU spends most of its time waiting on memory. A real inference service is almost always dominated by decode. , the token-by-token generation, is memory-bound: it waits on HBM High Bandwidth Memory. Memory stacked in layers and soldered right next to the GPU, with several TB/s of bandwidth (versus ~50 GB/s for DDR5). Indispensable beyond a certain model size. , the GPU’s high-bandwidth memory. Bolting the two onto the same GPU underuses the hardware on one of the two phases. NVIDIA acknowledged this at the silicon level by integrating a dedicated Groq LPU for decode into the Vera Rubin platform, and by launching Dynamo 1.0 as the orchestrator.
A week later, the thesis is no longer an architectural projection. Three software stacks offer disaggregation in production. A hardware challenger disputes the principle itself. And the first MLPerf cycle of the disaggregation era measures what software is worth against silicon. This is the state of play this article surveys.
The disaggregated pipeline in one diagram
The difficulty of this pipeline lives in the third column. The KV cache The stored key and value vectors an LLM has already computed for every token it has processed. It avoids recomputing attention over the whole history, at the cost of a memory footprint that grows with the context length. , the attention keys and values the model computes from the prompt and reuses for every later token, is produced by the prefill node and must reach the decode node before generation can start, and that transfer is measured in gigabytes per request. On a 70B model in FP8 An 8-bit floating-point format. A versatile working format for inference (and training) on recent GPUs. It halves the memory footprint and bandwidth needed versus FP16, for a marginal accuracy loss on most models. , an 8k-token context amounts to ~1.3 GB of KV cache. On a DeepSeek-R1 671B with a 100k context, it runs to tens of gigabytes. If the transfer is slower than decode itself, disaggregation degrades latency instead of improving it. Each runtime solves this its own way.
Dynamo 1.0: the orchestrator in production
NVIDIA shipped Dynamo 1.0 as a production release on March 16, 2026, the same day as the Vera Rubin announcement. That timing is deliberate: Dynamo is the software link between the hardware thesis (silicon specialized per phase) and its execution.
What Dynamo does. It is an inference orchestration framework, open source (GitHub: ai-dynamo/dynamo), that coordinates a cluster of prefill and decode nodes. It routes requests in real time, transfers the KV cache between nodes, and manages the elasticity of the pool. Its native mode is disaggregated serving: the prefill nodes compute the prompt, the KV cache is transferred over NVLink NVIDIA's proprietary GPU-to-GPU interconnect. NVLink 5 (Blackwell) reaches 1.8 TB/s per GPU; NVLink 6 (Rubin) doubles that to 3.6 TB/s. It lets several cards share their memory and behave almost like a single accelerator. (NVIDIA’s chip-to-chip interconnect) or RDMA to the decode nodes, and generation starts without the prefill node being blocked.
What sets it apart from the runtimes. Dynamo is not an inference runtime: it relies on TensorRT-LLM or SGLang as the execution backend. Its role sits above: KV-aware routing, topological placement across the cluster, and above all DGDR (Dynamo Graph Deployment Request), a mechanism that translates SLOs (target latency, minimum throughput) into a concrete deployment on the cluster. Where a runtime answers “how do I serve this model on this GPU”, Dynamo answers “how do I spread this model across 72 GPUs to hold these constraints”.
Multimodal E/P/D. Since March 2026, Dynamo extends the model to three phases (Embedding, Prefill, Decode) with a dedicated embedding cache. On image requests with Qwen3-VL-30B-A3B-Instruct in FP8 on GB200, NVIDIA measures a 30% reduction in TTFT (time to first token) and a 25% throughput gain (verified fact, NVIDIA source). The extension to video is underway: FastVideo plus SGLang Diffusion generates 5 seconds of video in ~40 seconds on a single Hopper GPU.
Kubernetes integration. The K8s Inference Gateway plugin implements KV-aware routing directly inside Kubernetes infrastructure: requests that share a KV prefix are routed to the node that already holds it. NVIDIA measures a 20× faster TTFT and a 4× faster end-to-end latency on requests with a common prefix (verified fact). Grove (GitHub: ai-dynamo/grove) rounds out the picture with topology-aware scheduling for GB300 NVL72: it places workloads according to the rack’s NVLink topology.
Adoption. The list of confirmed adopters as of May 24, 2026 includes AstraZeneca, Baseten, ByteDance, CoreWeave, Crusoe, DigitalOcean, Gcore, GMI Cloud, Nebius, Meituan, Pinterest, Prime Intellect, Rednote, SoftBank Corp., Tencent Cloud, Together AI, and Vultr. Cloud integrations: AWS, Azure, GCP, OCI, Alibaba Cloud.
The Baseten case: the nuanced numbers
Baseten published a field report as early as October 2025, as an early adopter. The post’s title (“How Baseten Achieved 2x Faster Inference with NVIDIA Dynamo”) sums up a TTFT cut in half. The detailed measurements, by Abu Qader, Michael Feil, and Rachel Rapp, are more granular: a 50% reduction in TTFT, a 61% increase in requests per second, and a 62% increase in tokens per second on Qwen3 Coder 480B with prompts of about 50k tokens. The “2×” applies to TTFT, not to overall throughput. And the “no additional hardware cost” claim comes from social media, not from the technical post.
The aggregate throughput figure NVIDIA cites (7× throughput with disaggregation plus extended Expert Parallelism on GB200 NVL72, measured on DeepSeek R1-0528 in FP4 A 4-bit floating-point format, the 2026 frontier of high-throughput inference. Four times less memory than FP16, but a very narrow dynamic range: it only holds up with fine-grained scaling through block formats (MXFP4, NVFP4). , 1k/1k context) comes from a SemiAnalysis InferenceX benchmark dated March 3, 2026, relayed by NVIDIA. It is a vendor-cited figure: representative of an optimized case, not of an average production throughput.
SGLang 0.5.12: native disaggregation
SGLang 0.5.12 shipped on May 16, 2026, and it is the first version of the runtime to treat prefill/decode disaggregation as a first-class citizen. Where Dynamo is an orchestrator above the runtime, SGLang builds the machinery into the runtime itself.
The mechanism. The GPU Staging Buffer is an intermediate VRAM buffer that accumulates the KV cache on the prefill side and transfers it block by block to the decode node. The Decode Radix Cache on the decode side organizes the received KV in a radix tree that enables prefix matching: if two requests share the same first 2,000 tokens, the cache is transferred only once. Three transport backends are supported: NIXL (NVIDIA), Mooncake (an open-source framework), and MORI-IO (AMD), which makes SGLang the only runtime to natively support disaggregation on both GPU ecosystems.
DeepSeek V4 day-0. SGLang 0.5.12 is the first runtime to support the full DeepSeek V4 inference path from launch day. This matters: DeepSeek models use a MoE (Mixture of Experts) architecture with a very large number of experts, and serving them efficiently demands Tensor Parallelism, Expert Parallelism, Context Parallelism, and Data Parallel Attention at once. SGLang supports TP16 on H100 and H20.
HiSparse: proactive offload of idle KV. HiSparse is a mechanism specific to sparse-attention models of the DeepSeek family (V3.2, GLM-5.1). Instead of waiting for VRAM to saturate before starting the offload, HiSparse proactively identifies the KV entries that no longer participate in attention and moves them to CPU memory. The measured gain: 3× throughput at 256 concurrent requests, up to 5× on long contexts. The limit: it works only on sparse-attention models (DSA). On a classic dense-attention Llama, the mechanism does not apply.
HiCache complements HiSparse by adding a unified radix tree (UnifiedRadixTree) that tiers the KV cache across VRAM, CPU RAM, and SSD via Mooncake. It extends the logic of PagedAttention, which pages the KV cache the way an operating system pages virtual memory, to three storage levels instead of two.
Native Blackwell. The TokenSpeed MLA backend is optimized for Blackwell Tensor Cores with an FP8 KV cache. The W4A4 MegaMoE kernels target 4-bit-quantized MoE models, and speculative decoding A decode-acceleration technique that uses a small fast model (draft model) to propose several candidate tokens, then verified in a single pass by the main model. Correct tokens are accepted at no extra cost: you trade redundant compute for fewer autoregressive passes. V2, where a lightweight draft path proposes several tokens that the full model then verifies in one batch, improves the acceptance rate on MTP (Multi-Token Prediction) models. DeepSeek-R1 ships a native MTP layer, extended to 3 for the performance peak in Dynamo plus TRT-LLM.
TensorRT-LLM 1.3: Helix Parallelism and the KV connectors
TensorRT-LLM 1.3 has been in a fast pre-release cycle since March 2026: rc7 on March 10, rc12 on April 17, rc15 on May 21. Each release candidate adds a piece of the disaggregation puzzle.
Service discovery for disaggregated serving (rc12): prefill and decode nodes discover each other automatically on the cluster, with no static configuration. It is the prerequisite for Dynamo to route requests without a hard-coded registry.
KV Cache Connector API (introduced in v1.1, refined in the 1.3 release candidates): an abstraction API for KV transfer between nodes. The runtime does not need to know how the KV is transferred (NVLink, RDMA, S3/Azure blob): the connector handles it. Dynamo uses this API to offload KV to object storage, a feature shipped in March 2026.
Sparse Attention (rc13): support for MQA and GQA sparse attention, needed for DeepSeek models and sparse-attention architectures.
Helix Parallelism: the central mechanism
Helix Parallelism is TRT-LLM 1.3’s most significant contribution to disaggregated serving. Its mechanism deserves the detail, because it solves a problem that existing parallelism strategies handled poorly.
Here is the problem. On a MoE model like DeepSeek-R1 671B, each transformer layer consists of two blocks: attention (which is dense, every parameter participates) and the FFN (feed-forward network, which is sparse, only a few experts fire per token). Classic Tensor Parallelism splits all operations uniformly across the GPUs. On dense attention, that is optimal: each GPU does its share. On the sparse FFN, it is waste: the GPUs hosting inactive experts sit idle, while those hosting the hot experts saturate.
Helix resolves this imbalance by applying two different parallelism strategies depending on the block. During attention, it uses KV Parallelism: each GPU holds a partition of the KV cache and computes its share of attention independently. During the FFN, it switches to TP (Tensor Parallelism) for dense models, or TP×EP (Tensor Parallelism × Expert Parallelism) for MoE models. This dynamic switching between two parallelism regimes within a single layer (hence the name Helix, which evokes interleaving) is what sets the approach apart. The paper (arXiv: 2507.07120) measures up to a 1.5× reduction in TTL (Token-to-Token Latency) and the ability to support 32× larger batches under the same latency constraint, on DeepSeek-R1 on Blackwell.
It is a software answer to the same problem that hardware disaggregation attacks on the silicon side: match the execution regime to the workload profile, layer by layer.
RocketKV: compression without training
RocketKV (arXiv: 2502.14051, ICML 2025, NVlabs) is a two-stage KV compression technique integrated into TRT-LLM 1.3. The first stage, SnapKV++, evicts the KV entries that contribute little to attention, by measuring their accumulated importance score. The second stage applies a hybrid top-k sparse attention on the remaining entries. The gain measured on H100: a 3× decode speedup, a 31% reduction in peak memory use. The distinctive point: no retraining is required. The compression applies at inference, on an existing model.
The counter-argument: Tenstorrent Galaxy Blackhole
All the solutions above start from one premise: disaggregation is necessary because the general-purpose GPU is structurally sub-optimal on one of the two phases. Tenstorrent disputes that premise.
On April 28, 2026, Tenstorrent launched Galaxy Blackhole into general availability. The positioning is explicit in their press release: “Other solutions require bolting together separate accelerators across fragmented infrastructure.” Their thesis: rather than fragmenting the pipeline across different silicon linked by a network, you unify compute, memory, and network in a single system built so that the internal bandwidth is high enough that the prefill/decode distinction loses its relevance.
The specs behind the thesis. Galaxy Blackhole bundles 32 Blackhole chips into a single system. Compute reaches 23 PFLOPS in Block FP8. Memory is split across two tiers: 6.2 GB of SRAM at 2.9 PB/s of internal bandwidth, so that the SRAM alone offers a bandwidth on a different scale from the HBM of a classic GPU. For reference, an H100 SXM5 tops out at ~3.35 TB/s of HBM3, a B200 at ~8 TB/s of HBM3E. Above that sits 1 TB of DRAM at 16 TB/s. The internal network supports up to 56 ports of 800G Ethernet.
Tenstorrent’s reasoning runs as follows. The reason decode is memory-bound on a classic GPU is that HBM bandwidth (~8 TB/s on a B200) cannot keep pace with the compute. If you replace the GPU-plus-HBM hierarchy with an SRAM-first hierarchy at 2.9 PB/s, the arithmetic intensity of decode rises enough that compute becomes the limiting factor again, not memory. The result: the prefill/decode split loses its purpose. A single system does both efficiently.
Blitz Mode makes this thesis concrete. On DeepSeek-R1-0528 (671B parameters), Tenstorrent claims: 350+ tokens/s/user in decode, TTFT under 4 seconds on a 100k-token context, batch sizes from 8 to 64, context up to 128k. These are vendor figures (a verified fact as to their publication, not verified by an independent benchmark), but the sub-4-second TTFT on 100k tokens is an aggressive claim: it means the prefill itself is fast despite the absence of dedicated Tensor Cores in NVIDIA’s sense of the term.
The price. A Galaxy Blackhole system starts at around $110,000. A 4-system supercluster starts at around $440,000. Set against the cost of a DGX GB200 NVL72 rack (press estimates put it between $2 million and $3 million), that is an order of magnitude below. The direct comparison stays misleading, because the two systems do not target the same model scale or the same inter-rack network bandwidth.
AMD: MORI-IO and vLLM-ATOM
AMD has no orchestration framework equivalent to Dynamo. Its strategy is to supply the transport building blocks that third-party runtimes consume, and to make sure its GPUs are first-class citizens in SGLang and vLLM.
MORI-IO (Modular RDMA Interface) is AMD’s point-to-point communication library for low-overhead KV transfers (GitHub: ROCm/mori). It is not an inference runtime: it is an RDMA transport layer optimized for disaggregation use cases, namely the transfer of KV cache between prefill and decode nodes (the Expert Parallelism communication of MoE models falls to a separate module, MORI-EP). MORI-IO is integrated into SGLang (as a transport backend alternative to NIXL) and into vLLM. It is the piece that makes disaggregation possible on AMD GPUs, exactly as NIXL makes it possible on NVIDIA.
vLLM-ATOM is an external plugin published on May 7, 2026, that bundles AMD’s kernel optimizations for vLLM: AITER fused attention, custom AllReduce, optimized MoE routing, FP4 support for MI355X. AMD describes it as a “temporary proving ground for new optimizations”; the stabilized kernels are upstreamed into vLLM’s native ROCm backend. The logic is pragmatic: rather than wait on upstream, AMD ships the optimizations in a plugin you install alongside, one that will disappear once upstream has absorbed them.
MI350P PCIe deserves a mention. Announced on May 7, 2026, it is the first AMD MI350 accelerator in PCIe Gen 5 form factor: 144 GB of HBM3E at 4 TB/s, up to 4.6 PFLOPS in MXFP4, passive dual-slot, 600 W TBP (configurable down to 450 W). It is not the same class as the MI355X (288 GB, 8 TB/s): it is a drop-in for existing air-cooled servers, designed for companies that want to enter disaggregated inference without overhauling their cooling infrastructure.
What MLPerf v6.0 measures
MLPerf Inference v6.0, published on April 1, 2026 by MLCommons, is the first benchmark cycle that reflects the era of software disaggregation. 24 organizations submitted results. Five new benchmarks were added: GPT-OSS 120B, interactive DeepSeek-R1 (with speculative decoding), DLRMv3, WAN-2.2 text-to-video, and YOLOv11 edge.
The most striking result is not a hardware number: it is a software number. NVIDIA improved its DeepSeek-R1 score by 2.7× over its own earlier submission. The hardware did not change: 288 GPUs, 4 GB300 NVL72 racks. The gain comes from orchestration: Dynamo, disaggregation, Expert Parallelism, extended speculative decoding (MTP raised from 1 to 3 layers). The result: 2.5M tokens/s. This 2.7× has to be read precisely: it is against the earlier NVIDIA submission, not against the whole MLPerf v5.1 field. It shows that software weighs as much as silicon on already-deployed hardware.
AMD MI355X. The flagship score is 100,282 tokens/s on Llama 2 70B Server, 3.1× the MI325X. In multinode (11 nodes, 87 MI355X), AMD reaches 1,042,110 tokens/s Offline and 1,016,380 tokens/s Server. These figures confirm the MI355X’s competitiveness in raw throughput, even if AMD’s software ecosystem stays a notch behind on orchestration maturity.
NVIDIA Blackwell Ultra reaches 2.5M tokens/s on DeepSeek-R1 with 288 GPUs spread across 4 GB300 NVL72 racks, the highest absolute score of the cycle. That this score comes largely from software gains on existing hardware is the most important signal of this MLPerf cycle: investing in orchestration pays as much as investing in silicon.
The implementations compared
| Criterion | Dynamo 1.0 | SGLang 0.5.12 | TRT-LLM 1.3 | Tenstorrent Galaxy |
|---|---|---|---|---|
| Role | Cluster orchestrator | Native runtime | Runtime + kernels | Integrated system |
| Disaggregation | Native (E/P/D) | Native (P/D) | Via Service Discovery + KV Connector | Not applicable (unified) |
| KV transport | NVLink, RDMA, S3/Azure blob | NIXL, Mooncake, MORI-IO | KV Cache Connector API | Internal SRAM (2.9 PB/s) |
| Supported GPUs | NVIDIA (GB200, GB300) | NVIDIA + AMD (H100, H20, MI355X) | NVIDIA only | Blackhole (RISC-V Tensix) |
| Advanced parallelism | Wide EP, TP, KV-aware routing | TP, EP, CP, DPA | Helix (KV-P + TP×EP) | Internal, 32 chips |
| KV optimization | KVBM offload S3, prefix caching | HiSparse, HiCache, Radix Cache | RocketKV, Sparse Attention | SRAM-first (6.2 GB) |
| Kubernetes | Inference Gateway + Grove | No | No | No |
| Open source | Yes (ai-dynamo/dynamo) | Yes (sgl-project/sglang) | Yes (NVIDIA/TensorRT-LLM) | No |
| Entry price | Cloud GPU (~$2-3/h H100) | Cloud GPU (~$2-3/h H100) | NVIDIA GPU only | ~$110,000 (system) |
How to choose
The decision matrix depends on three variables: your existing hardware fleet, your dominant constraint (latency or throughput), and your tolerance for operational complexity.
You have an existing NVIDIA fleet and you serve MoE models at high concurrency. Dynamo plus TRT-LLM is the most direct path. Dynamo orchestrates the disaggregation, TRT-LLM delivers the maximum per-node throughput with Helix Parallelism, and the Kubernetes plugin handles scaling. The operational cost is real (one more layer to maintain), but the measured gains (7× throughput on the NVIDIA benchmark, 2× TTFT at Baseten) justify the investment on high-concurrency workloads.
You want disaggregation without hardware lock-in. SGLang 0.5.12 is the only runtime that supports disaggregation natively on NVIDIA and AMD, via NIXL and MORI-IO respectively. If you plan a mixed fleet or a future migration to AMD, it is the most defensible choice. HiSparse is a powerful lever if you serve DeepSeek models.
You start from scratch and per-user latency is your key metric. Tenstorrent Galaxy Blackhole is worth the evaluation. The sub-4-second TTFT on 100k tokens and the 350+ tokens/s/user in decode are aggressive claims, but the entry price (~$110,000 per system) and the absence of KV-transfer complexity are real arguments. The trade-off: a young software ecosystem, no native Kubernetes, and a lock-in to the Tensix architecture.
You serve mostly dense models (Llama, Mistral) at moderate batch. Disaggregation is probably not your immediate priority. A well-configured classic runtime, vLLM with PagedAttention An algorithm introduced by vLLM that manages the KV cache like an operating system's virtual memory: in pages, without requiring a contiguous block per request. It eliminates internal and external fragmentation, letting a server handle 2 to 4× more concurrent requests on the same VRAM. and prefix caching, covers the majority of these cases. Disaggregation pays off mainly on MoE models at very high concurrency or on long contexts.
Conclusion
Software caught up with hardware in three months. The thesis Vera Rubin laid out, that inference breaks into specialized phases, is now implementable without waiting for Rubin silicon. Dynamo, SGLang, and TRT-LLM let you disaggregate on existing Blackwell or Hopper hardware, and the measured gains (MLPerf v6.0: 2.7× from software alone) show that orchestration has become a lever at least as powerful as the jump to the next silicon generation.
The next frontier is not disaggregation itself: it is its automation. The three runtimes still demand manual sizing of the prefill/decode ratio, explicit configuration of the KV transport, and a network topology designed for the use case. Dynamo’s DGDR sketches the direction: translate an SLO into a deployment, without the operator touching the routing. Once that loop closes, an SLO goes in and a configured cluster comes out, disaggregation will stop being a specialist’s technique and become a default deployment mode.
The other open question is the fourth tier. Vera Rubin laid out three tiers: prefill GPU, decode LPU, orchestration CPU. The KV cache, whose transfer is the bottleneck of disaggregation, could justify a dedicated storage tier, exactly what the cancelled Rubin CPX was meant to be. If KV transfers remain the limiting factor at scale, that tier will reappear, in one form or another, in the Feynman 2028 platform.
Sources and method
This article draws on a research file closed on May 24, 2026. Labels: verified fact = citable primary source; credible estimate = consistent but not independently verified; assumption = reasoning without measurement. A French version of this article, published on May 24, 2026, is the original.
Dynamo 1.0
- NVIDIA Developer Blog, NVIDIA Dynamo 1.0 Production Ready: developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/, March 16, 2026 (verified fact). Multimodal E/P/D figures, Kubernetes Inference Gateway, FastVideo.
- GitHub: ai-dynamo/dynamo, open source (verified fact).
- GitHub: ai-dynamo/grove, Grove API for topology-aware scheduling on GB300 NVL72 (verified fact).
- SemiAnalysis InferenceX, 7× throughput benchmark on GB200 NVL72 with DeepSeek R1-0528 FP4 (vendor-cited, relayed by NVIDIA).
- Baseten, “How Baseten Achieved 2x Faster Inference with NVIDIA Dynamo”: baseten.co/blog/how-baseten-achieved-2x-faster-inference-with-nvidia-dynamo/, October 16, 2025, Abu Qader, Michael Feil, Rachel Rapp. TTFT -50%, +61% RPS, +62% TPS on Qwen3 Coder 480B (verified fact). The “2×” claim applies to TTFT. The “no additional hardware cost” claim comes from social media, not from the technical post (editorial nuance).
SGLang 0.5.12
- GitHub release: sgl-project/sglang v0.5.12, May 16, 2026 (verified fact).
- PD disaggregation with GPU Staging Buffer, Decode Radix Cache, NIXL/Mooncake/MORI-IO backends (verified fact).
- HiSparse: 3× throughput at 256 concurrent requests, up to 5× on long contexts (verified fact, limited to DSA models: DeepSeek-V3.2, GLM-5.1).
- HiCache: UnifiedRadixTree, SSD offload via Mooncake (verified fact).
- DeepSeek V4 day-0, TP16 on H100/H20 (verified fact).
TensorRT-LLM 1.3
- Releases: rc7 (March 10) to rc15 (May 21, 2026), pre-release (verified fact).
- Helix Parallelism: arXiv 2507.07120, KV Parallelism (attention) + TP×EP (FFN), 1.5× TTL, 32× batch at constant latency on DeepSeek-R1/Blackwell (verified fact).
- RocketKV: arXiv 2502.14051, ICML 2025 (NVlabs), 3× decode speedup, -31% peak memory on H100 (verified fact).
- Service discovery (rc12), KV Cache Connector API (v1.1+), MQA/GQA Sparse Attention (rc13) (verified fact).
Tenstorrent Galaxy Blackhole
- Tenstorrent Newsroom, GA April 28, 2026: tenstorrent.com/newsroom/ (verified fact).
- Specs: 32 Blackhole chips, 23 PFLOPS Block FP8, 6.2 GB SRAM (2.9 PB/s), 1 TB DRAM (16 TB/s), 56× 800G Ethernet (verified fact).
- Blitz Mode: 350+ tokens/s/user decode, TTFT under 4 s on 100k context, DeepSeek-R1-0528 671B, batch 8-64, context up to 128k (verified fact as vendor publication, not verified by independent benchmark).
- Price: system starting at $110,000, 4-system supercluster starting at $440,000 (verified fact).
AMD
- ROCm Blog, vLLM-ATOM plugin: rocm.blogs.amd.com, May 7, 2026 (verified fact).
- MORI-IO (Modular RDMA Interface): github.com/ROCm/mori, RDMA library for KV and EP transfer on AMD GPUs (verified fact).
- MI350P PCIe: AMD blog May 7, 2026, 144 GB HBM3E, 4 TB/s, 4.6 PFLOPS MXFP4, dual-slot PCIe Gen 5, 600 W TBP (verified fact).
MLPerf Inference v6.0
- MLCommons, MLPerf Inference v6.0 Results: mlcommons.org/2026/04/mlperf-inference-v6-0-results/, April 1, 2026 (verified fact).
- NVIDIA: 2.5M tokens/s DeepSeek-R1, 288 GPUs, 4× GB300 NVL72 (verified fact). The 2.7× is against the earlier NVIDIA submission, not against the whole v5.1 field (editorial clarification).
- AMD MI355X: 100,282 tokens/s Llama 2 70B Server (3.1× vs MI325X), 1,042,110 tokens/s Offline / 1,016,380 Server on 11 nodes / 87 MI355X (verified fact).
Speculative decoding
- EAGLE-3 / SpecForge: arXiv 2603.18567, March 2026. Measured range: 3.27× to 4.79× on HumanEval (LLaMA-3.3-70B-Instruct, temp=0) (verified fact).
- P-EAGLE (AWS): March 13, 2026, main PR #32887. 1.55× at c=1, 1.05× at c=64 on MT-Bench/GPT-OSS-20B (verified fact).
Prices
All prices in this article are in US dollars, as reported by the cited sources (Tenstorrent’s published pricing, cloud GPU street rates). The French edition converts them to euros at an indicative 1 USD = 0.92 EUR (mid-2026).
