What DeepSeek has released
Downloading a model’s weights gives you the parameters learned during training. You still have to load them, organize the computation and distribute requests. Two teams serving the same model on comparable machines can therefore deliver very different response times and costs.
On October 20, 2026, DeepSeek released DeepInfer, which it describes as the engine behind its production services. The Apache 2.0 repository provides a specialized implementation for DeepSeek-V4.1-Flash. The examined version declares support for NVIDIA SM100 GPUs and Huawei Ascend 950 accelerators. Its initial public usage mode is offline inference: running requests without supplying a complete production API. Official documentation.
The release makes a question left open by model weights much easier to investigate: how do you turn chip capacity into useful responses? Its code exposes memory, networking and scheduling decisions, along with the boundary between the engine and the infrastructure around it.
A request passes through more than the model
You send an assistant a long document, then ask a second question about the same text. The service benefits from reusing some of the first request’s work. It must find that state, choose the machines that will handle the follow-up and move the data to the right place.
prefill The opening phase of LLM inference: prompt tokens are processed in parallel to build the context state. That reuse raises arithmetic intensity and can make the phase compute-bound; the exact regime depends on the model, batch, context and backend. processes the input tokens, the pieces of text handled by the model. decode The autoregressive generation phase of an LLM: one token is produced at a time while rereading weights and the relevant context state. At low batch sizes, that traffic can make decode memory-bandwidth-bound. Its share of total cost still depends on input and output lengths, batching and cache reuse. then generates the response incrementally. The KV cache The stored key and value vectors an LLM has already computed for every token it has processed. It avoids recomputing attention over the whole history, at the cost of a memory footprint that grows with the context length. retains intermediate attention information so that each step does not require recomputing the entire context.
The documentation explicitly identifies what a full service still needs: the KV cache system, the integration of separated computation phases and load balancing across replicas. A replica here is a group of machines capable of serving requests. Distributing requests among these groups is a separate operation from distributing a model’s experts within one group.
The separate deepseek-recipe project handles another boundary: converting requests and responses between API formats. It helps expose the model to applications, but its caller must provide the engine, transport and tool execution. This separation makes the components reusable without making their combination a ready-to-run service. Recipe documentation.
Experts need to work in the right places
DeepSeek-V4.1-Flash uses a mixture of experts, or MoE Mixture-of-Experts. An architecture where the network is split into many experts, of which a router activates only a small subset per token. Per-token compute follows the number of active parameters; memory follows the total count, since every expert must stay resident in VRAM, ready to be called. . For each token, it selects particular subnetworks instead of activating every available expert. Those experts can be spread across several accelerators. Each machine stores part of the weights, but data must reach the selected experts and return to the rest of the computation.
Suppose requests disproportionately activate the experts on one machine. Other machines may finish their work and wait for it. Adding cards does not automatically fix this imbalance. The software needs to observe demand, move experts or replicate the busiest ones.
DeepInfer publishes a placement and replication policy for this purpose. It is an important part of expert parallelism A way of splitting a MoE where each GPU holds a subset of the experts, instead of receiving a slice of every matrix as in tensor parallelism. Each token therefore has to travel to whichever GPUs hold the experts picked for it, so the collective becomes an all-to-all rather than an all-reduce. Usually shortened to EP. : the model’s distribution must match the distribution of actual work. Communication and weight reads become as important as multiplication speed. Expert load-balancing policy.
This explains why performance measured on a large GPU group cannot be transferred to an isolated card. Each accelerator belongs to a system whose placement, batches and interconnect affect the result. Our analysis of MoE, batching and networking develops that relationship between useful work and communication overhead.
Reduce the work before adding compute
The engine also uses the model’s architecture to avoid some computation. V4.1-Flash shares part of its attention information across layers. In certain decoder layers, DeepInfer can reconstruct the required state by replaying a bounded portion of the prompt’s tail instead of processing the entire sequence again in those layers.
That does not mean the model ignores the beginning of the document. Other parts of the architecture have already processed the earlier information. The technical report nevertheless describes approximately reconstructed states and adaptation during post-training. This optimization therefore needs to be assessed with the model’s quality evaluations, alongside its token throughput. V4.1-Flash report, section 3.2.2.
Another mechanism, speculative decoding A decode-acceleration technique that uses a small fast model (draft model) to propose several candidate tokens, then verified in a single pass by the main model. Correct tokens are accepted at no extra cost: you trade redundant compute for fewer autoregressive passes. , uses a draft model to propose several tokens before the main model verifies them. Accepted proposals advance several positions together. Rejected proposals mean some drafting and verification work has not advanced the response.
DSpark uses a confidence estimate to choose how many proposals to verify. A request with unpromising drafts may not deserve as much space in the verification batch as another request. The choice also depends on how the hardware processes that batch. The release makes this coordination inspectable; the method itself predates DeepInfer. DSpark paper.
Meanwhile, the CPU prepares batches while the accelerators compute. Overlapping those tasks reduces the gaps in which GPUs wait for the next software decision. This remains useful with very fast kernels: shortening a compute step makes the delays between steps a larger share of elapsed time. Engine scheduler.
A million tokens per second needs a denominator
One point on the official generation curve implies roughly one million tokens per second for a group of 64 SM100 GPUs. That total comes from approximately 15,700 tokens per second per GPU, read from the chart. It is neither one user’s speed nor the throughput of a standalone card. Official curve.
At the same point, the curve shows about 246 tokens per second per user, with 64 requests per compute rank. The protocol uses a 131,072-token context, three speculative proposals and a supplied acceptance rate of 75%. The plotting code converts compute steps into tokens using that acceptance assumption. The reported throughput must be read with this condition attached.
The scripts add further conditions: random tokens, forced expert load balancing, warm-up and reuse of a previously computed prefix. They measure a steady operating state. They do not by themselves reproduce irregular client arrivals, cache misses, queues and interruptions in a service. Protocol and scripts.
| Metric | What it tells you | What still needs measuring |
|---|---|---|
| Tokens per second for the group | Text produced under the tested load | Throughput under a user latency requirement |
| Tokens per second per GPU | Group throughput divided by card count | Performance of a smaller configuration |
| Tokens per second per user | Individual speed at the chosen concurrency | Time to first token and tail slowdowns |
These distinctions also matter when comparing vLLM or SGLang. Changing topology, cache hit rate or response length can change the ranking. The repository supplies a technical reference; a general ranking of engines would require a shared protocol.
Engine throughput does not measure the request queue
The curve exposes a practical trade-off. Toward its left, more requests share the resources: the group produces more tokens overall, but each user progresses more slowly. Toward the right, individual speed takes priority and the system delivers less text per second in total. The best operating point therefore depends on the application’s acceptable delay, alongside throughput.
A request may also wait before entering the measured batch. For an interactive assistant, that delay matters as much as generation speed: a fast response once started may still arrive too late if a queue has built up. Adding requests to a benchmark and increasing the arrival rate of clients are different experiments.
One useful deployment evaluation would keep the same prompt set and progressively increase its arrival rate. It would record time to first token, speed during the response and the proportion of requests meeting the latency target, separating reused prefixes from those that need recomputing. This proposed protocol would complement offline measurements; no such result is claimed here.
Token cost remains a service-level question
An input token found in the cache does not require the same work as a new token to process. An output token adds another kind of load. Summing those categories describes a service volume, but does not measure compute capacity or economic efficiency on its own.
Calculating a cost means dividing system spending by the work actually delivered. Accelerators continue to incur costs between requests. Cache storage, network interfaces, cooling and spare capacity also have a price. A configuration that maximizes throughput can be unacceptable if users have to wait too long.
API pricing adds a commercial decision to that accounting. It does not directly reveal technical cost, and releasing an engine does not establish the cause of a price reduction. The code exposes mechanisms that can save resources; a complete service measurement is still needed to quantify their financial effect.
DeepInfer’s immediate value is practical: a team can inspect DeepSeek’s decisions, test an optimization and compare its results with a specialized reference. The project acknowledges its origins in vLLM and contributions from SGLang and TensorRT-LLM. The release returns implementation choices to a software ecosystem from which DeepSeek has also benefited.
For an operator, the useful next step is to compare the time needed to serve the same requests at the same quality and latency targets. Opening the engine makes that investigation more concrete. The service around it still needs to be built.
Sources and method
Analysis as of November 1, 2026, based on LeCompute’s research dossier, the code and the linked publications. DeepInfer reference: commit 553741557a57df44a46fb3ab72b6aafeaf945337. No original benchmark is claimed.
- Documented facts: engine README and license, PyPI 0.1.0 release and Recipe. Production use is DeepSeek’s statement; the complete identity of public and deployed versions has not been independently audited.
- Inspectable mechanisms: local cache, expert policy and scheduler linked in the body. V4.1-Flash and DSpark document the methods and their conditions.
- Reported measurements and estimate: the decode curve supplies approximate values read from an image.
15,700 × 64 = 1,004,800is rounded to one million to avoid false precision. The 75% acceptance rate is a plotting parameter, not a measurement on readers’ conversations. - Analysis: implications for service quality and costs are conditional reasoning. No actual cost per token, universal speedup or NVIDIA/Huawei parity result is inferred from these curves.
