A collective costs more than its bytes
An MoE Mixture-of-Experts. An architecture where the network is split into many experts, of which a router activates only a small subset per token. Per-token compute follows the number of active parameters; memory follows the total count, since every expert must stay resident in VRAM, ready to be called. router selects a small subset of experts for each token. Compute follows active parameters, while every expert’s weights still need a home. expert parallelism A way of splitting a MoE where each GPU holds a subset of the experts, instead of receiving a slice of every matrix as in tensor parallelism. Each token therefore has to travel to whichever GPUs hold the experts picked for it, so the collective becomes an all-to-all rather than an all-reduce. Usually shortened to EP. places a subset of experts on each GPU. Tokens travel to the devices holding their selected experts, and the resulting activations travel back.
That path creates two all-to-all A collective where every rank sends distinct data to every other rank. An expert-parallel MoE layer runs two of them: dispatch ships each token's hidden state to the GPUs holding its experts, combine brings the outputs back. Its volume scales with the token count, so unlike weight reads it never amortizes over a larger batch. collectives per MoE layer: dispatch and combine. Message volume scales with token count. Our MoE network crossover analysis shows why weight reads can dominate those bytes at low batch. DWDP targets something else: the rendezvous.
In a deployment combining data-parallel attention with partitioned experts, each rank serves different requests. Sequence lengths, cache-hit rates and selected experts diverge. One rank finishes early, but the next collective waits for the slowest. NVIDIA’s paper reports roughly 12 percent synchronization overhead at a 20 percent coefficient of variation in per-rank sequence length for its DeepSeek-R1 case. The accompanying TensorRT-LLM blog states roughly 10 percent for a nearby setup. That version difference is reason not to treat the number as a system constant. The mechanism does not depend on one percentage.
Smaller messages do not remove the rendezvous. Better expert balance can narrow compute skew without aligning request lengths or prefix-cache hits. DWDP asks a stronger question: can each rank progress without waiting for another rank’s tokens?
Reverse the path and pull experts to each rank
DWDP stands for Distributed Weight Data Parallelism. Attention remains data-parallel, so every GPU owns a request batch. Attention weights are replicated. Expert weights, which dominate a large MoE’s memory footprint, remain partitioned across a DWDP group. Before each MoE layer, a rank keeps its local experts and pulls the missing experts for that one layer from peer GPUs. It then computes every expert selected by its own tokens locally, with no token dispatch or combine.
cudaMemcpyAsync sends the transfer through GPU copy engines rather than SMs. Two buffers alternate: one feeds current compute while the other receives the next layer. The runtime primes early layers at the start of a forward step, then records one layer’s completion and launches the next prefetch. SGLang assembles composite virtual addresses with CUDA VMM. TensorRT-LLM exchanges CUDA IPC handles at startup and retains pointers to remote tensors.
The design does not permanently replicate the entire expert set on every GPU. Each rank stores its assigned experts plus a staging buffer for one upcoming layer. Full replication would reduce DWDP to ordinary data parallelism and remove its capacity advantage.
Moving more bytes can take less critical-path time
TensorRT-LLM’s roofline A model that bounds achievable performance with two ceilings: compute throughput and memory bandwidth. The ratio between them sets a crossover arithmetic intensity, below which a kernel is memory-bound and leaves its compute units idle, above which it is compute-bound. model compares two layer-level intervals. T_compute is the work available to cover a copy. T_prefetch is the time required to pull remote experts. When compute is shorter, the rank stalls for its staging buffer. Once compute lasts longer, much of the transfer leaves the critical path.
| Input length | Compute / prefetch | DEP time / DWDP time |
|---|---|---|
| 1,024 tokens | 0.19 | 0.10 |
| 8,192 tokens | 0.62 | 0.73 |
| 16,384 tokens | 1.52 | 1.27 |
| 32,768 tokens | 4.77 | 1.17 |
At 1K, this analytical case makes DWDP ten times slower because transfer overwhelms compute. At 16K, the window becomes large enough and DWDP moves 27 percent ahead. At 32K, prefetch hides more completely, yet speedup falls to 17 percent. Once compute dominates both strategies, removing communication subtracts a smaller share of total latency. Benefit is not monotonic with context length.
The 16K crossover is not portable. It fixes batch 1, DeepSeek-R1, GB200 and one partition. Higher batch enlarges compute and can make DWDP useful on shorter prompts. Replacing NVLink NVIDIA's proprietary GPU-to-GPU interconnect. NVLink 5 (Blackwell) reaches 1.8 TB/s bidirectional per GPU; NVLink 6 (Rubin) doubles that to 3.6 TB/s. With NVSwitch, it accelerates GPU-to-GPU transfers and collectives; memory remains physically distributed, and remote access does not have the cost of local HBM. with a slower fabric pushes the threshold in the other direction and may erase the winning regime.
SGLang’s 1.92× is real and prefill-only
SGLang 0.5.17, released August 8, 2026, ships DWDP as an early-development prefill The opening phase of LLM inference: prompt tokens are processed in parallel to build the context state. That reuse raises arithmetic intensity and can make the phase compute-bound; the exact regime depends on the model, batch, context and backend. feature. The merged pull request measures four B200 GPUs with gpt-oss-120b. DEP4 combines four-way expert parallelism and all-gather; DWDP4 prefetches weights over P2P. MNT is the maximum token count processed in one forward pass, and ISL is input sequence length.
| MNT | ISL 4K | ISL 8K | ISL 16K | ISL 32K |
|---|---|---|---|---|
| 16K | 1.16× | 1.37× | 1.18× | not reported |
| 32K | 1.30× | 1.45× | 1.48× | 1.92× |
At saturation, 128 concurrent requests and 8K input, SGLang reports 506,000 tokens/s against 329,000, or 1.54×. These results validate the implementation and the long-prefill regime. They do not include generation, queueing or user-perceived time to first token. Turning “1.92× prefill-only” into “1.92× faster MoE serving” would delete most of the protocol.
TensorRT-LLM’s trace shows where the missing gain went
On four GB200 GPUs, DeepSeek-R1, 8K input and 32,768 maximum tokens per forward pass, TensorRT-LLM breaks down one context iteration. DEP pays 126.74 microseconds of communication and 161.85 microseconds of synchronization. DWDP removes both from the critical path and overlaps 429 microseconds of P2P copy.
| Category | DEP4 | DWDP4 |
|---|---|---|
| Attention | 269.67 µs | 320.56 µs |
| Grouped GEMM | 342.40 µs | 337.42 µs |
| Critical communication | 126.74 µs | 0 µs |
| Synchronization | 161.85 µs | 0 µs |
| Overlapped P2P copy | 0 µs | 429.00 µs |
| Iteration | 1,319.85 µs | 1,131.58 µs |
Removing communication and synchronization promised a 21.86 percent gross reduction. Realized improvement is 14.26 percent. Attention rises from 269.67 to 320.56 microseconds, and other kernels slow too. Copy engines consume no SM slots, but their traffic still crosses the network-on-chip, L2 and DRAM on source and destination GPUs. Copy and compute contend in that hierarchy. NVIDIA’s follow-up identifies power-induced frequency throttling as another contributor.
A second conflict appears when several ranks pull experts from the same source GPU. Its copy engine serializes many-to-one requests. NVIDIA tests slicing transfers into 1 MB chunks and round-robin scheduling them across destinations, gaining another 8 percent in one short-window case. The blog states that this mitigation is not in the current productized path. It is a research result, not available performance.
Throughput per GPU rises while TTFT gets worse
The end-to-end experiment supplies the decisive adversarial control. TensorRT-LLM separates prefill and decode The autoregressive generation phase of an LLM: one token is produced at a time while rereading weights and the relevant context state. At low batch sizes, that traffic can make decode memory-bandwidth-bound. Its share of total cost still depends on input and output lengths, batching and cache reuse. workers, holds the generation server fixed and applies DWDP to context. The protocol uses DeepSeek-R1 NVFP4, 8K input, 1K output and compares Pareto points at nearby per-user throughput.
| Per-user speed | TPS/GPU gain | Baseline TTFT | DWDP TTFT |
|---|---|---|---|
| 20 to 30 tokens/s | 1.10× | 2,538 ms | 8,314 ms |
| 40 to 50 tokens/s | 1.08× | 1,919 ms | 7,012 ms |
| 60 to 70 tokens/s | 1.12× | 965 ms | 1,640 ms |
| 80 to 90 tokens/s | 1.06× | 1,669 ms | 2,280 ms |
| 170 to 180 tokens/s | 0.97× | 494 ms | 660 ms |
Across 20 to 100 tokens/s/user, the paper summarizes output-throughput efficiency at 8.8 percent more TPS/GPU. The table exposes what that average omits. DWDP Pareto points use fewer context GPUs, lower the context stage’s aggregate service rate and worsen rate matching with generation. In the 20 to 30 range, median TTFT moves from 2.5 seconds to 8.3. At the highest user speed, DWDP loses 3 percent per GPU.
This does not invalidate DWDP. It invalidates the single-metric verdict. An operator optimizing GPU count may accept a later first token to serve the same output rate with fewer context workers. An interactive product with a TTFT SLO Service Level Objective. A quality target a system sets for itself, say a time to first token under 500 ms at the 99th percentile. Distinct from an SLA, which is the contractual commitment you can enforce: in LLM inference, SLOs are everywhere and latency SLAs almost nowhere. may reject that exact point. Latency against latency explains why throughput, first-token delay and streaming fluidity cannot stand in for one another.
A narrow, useful validity domain
Long prefill or a sufficiently large batch. This is the intended regime. Current-layer computation covers next-layer weight transfer. Long RAG prompts, document ingestion and loaded context servers match the mechanism.
Disaggregated prefill/decode serving. TensorRT-LLM enables its productized path only on context workers. SGLang recommends disaggregated prefill too. Production disaggregation isolates the phase that owns the required compute window.
Fast intra-node P2P fabric. The paper targets GB200 NVL72; SGLang measures four NVLink-connected B200s. TensorRT-LLM’s CUDA IPC handles do not support cross-node deployment. A PCIe or RDMA cluster does not inherit the reported gains from the same flag.
Short decode or low batch. The wager reverses. Few tokens create little compute per layer while remote expert weights retain their size. The roofline table places DWDP behind DEP at 1K and 8K, batch 1.
A constrained software path. At publication, TensorRT-LLM requires the CuteDSL MoE backend with NVFP4, TP equal to 1 inside a DWDP group, the disaggregated MPI launch flow and fused-finalize FC2. It does not support overlap scheduler or EPLB on that path. SGLang calls its integration early-development. Those constraints define the product, not a footnote.
Conclusion
DWDP does not discover that expert weights are small. It discovers that a large transfer launched early can cost less critical-path time than a smaller synchronized transfer. The inversion works when prefill creates a compute window, NVLink delivers the next layer before use and every rank progresses without waiting for the slowest.
This is more than a kernel optimization. It changes the contract between data and model. Expert parallelism fixes weights in place and moves requests. DWDP fixes requests in place and circulates one layer of weights. Neither contract wins outside its regime. The first fits cheap token movement; the second fits a system where avoidable barriers cost more than hideable bytes.
The credible next step is not another prefill-only record. It is a policy that chooses the contract by workload, predicts copy time, protects TTFT and falls back to all-to-all as the compute window closes. Until that choice becomes automatic and extends beyond an NVLink island, DWDP remains specialized. That is also why systems engineers should understand it now.
Sources and method
The TensorRT-LLM mechanism and results come from NVIDIA’s paper DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72 and the TensorRT-LLM technical blog with reproduction inputs. The latter publishes the roofline, kernel profile, code constraints, end-to-end Pareto points and examples/dwdp/ reproduction files.
SGLang results come from release 0.5.17 and merged PR #29778. The pull request calls the code early-development, documents CUDA VMM and double buffering, and reports prefill-only measurements on four B200s. We preserve that qualification.
SGLang and TensorRT-LLM figures remain in separate tables because they change model, exact hardware, runtime, dataset and metric. No average or direct ratio is computed. The 1.92× figure is a SGLang prefill-only maximum; 8.8 percent is TensorRT-LLM’s serving summary across 20 to 100 tokens/s/user.
We found no independent replication of these DWDP measurements at publication. Both implementations derive from the NVIDIA concept, and their development teams produced the data. The code paths and TensorRT-LLM reproduction scripts are inspectable, but generalization to another MoE, fabric or request distribution remains a hypothesis to test.
