Three engines, three opposite defaults
Take the same model, the same GPU, the same workload. Serve it with vLLM, then SGLang, then TensorRT-LLM, without touching a single setting. You will get three different behaviours, and not because the kernel code differs. Because the three teams made opposite decisions on the same question.
vLLM enables chunked prefill Splitting an incoming prompt into pieces slotted into successive iterations instead of swallowing it whole. Stops one long prompt from freezing every conversation already in flight, at the cost of processing slightly less efficient than a contiguous prefill. Not to be confused with prefill-first, which prioritizes whole prefills. by default and gives it a budget of 2,048 tokens per iteration. Its documentation does not hide what that aims at: the value “is optimized for ITL Inter-Token Latency. The gap between two consecutive tokens of one response, which the user perceives as how smoothly the stream flows. Distinct from TPOT, which divides total decode time by the number of tokens produced and is therefore an average. Measurement trap: some harnesses fold the first token into the calculation and others exclude it, so their numbers are not comparable. , and it may have lower throughput than the default scheduler”. In other words, it protects the gap between successive tokens, even if that means serving fewer people.
SGLang enables the same chunking, but with 8,192-token pieces, four times larger. The project maintainer put it in writing in an August 2024 GitHub discussion: “We pick chunk size 8192 to favor throughput.” The opposite decision, owned and argued.
TensorRT-LLM does not enable it at all. The parameter exists, its default is False, and it is not even exposed as an option on the serving command: you have to edit a Python file to reach it. Its default capacity policy, GUARANTEED_NO_EVICT, guarantees that an admitted request will never be paused, even if that means forming smaller batches than memory would allow. Neither throughput nor inter-token time: what it protects is predictability.
Three competent teams, one problem, three defaults pointing in different directions. The lazy explanation would be that two of them are wrong. The real one is more interesting: they are not arbitrating the same thing, and the trade-off they are setting is not the one you think.
This is not a throughput-against-latency trade-off
The common intuition fits in a sentence: the more requests you group, the more people you serve, the longer each one waits. Throughput and latency would pull in opposite directions, and tuning an inference server would amount to placing a slider between them.
That description is false as a general law, and the refutation comes from the field’s founding technique. continuous batching Iteration-level scheduling: adding and removing requests from the batch at every generation step, instead of waiting for a whole batch to finish. Formalized by Orca (OSDI 2022), popularized by vLLM. It multiplies an inference server's throughput by 2 to 4× under heavy concurrency. , which lets requests enter and leave a batch at every iteration instead of waiting for the whole batch to finish, was introduced in June 2023 by an Anyscale post whose title says everything: twenty-three times the throughput while reducing median latency. The published measurements show an improvement at every percentile, at one request per second as at four. Not a slider moved: both metrics advance together.
This is not an isolated case. PagedAttention An algorithm introduced by vLLM that manages the KV cache like an operating system's virtual memory: in pages, without requiring a contiguous block per request. It eliminates internal and external fragmentation, letting a server handle 2 to 4× more concurrent requests on the same VRAM. , by managing the KV cache The stored key and value vectors an LLM has already computed for every token it has processed. It avoids recomputing attention over the whole history, at the cost of a memory footprint that grows with the context length. in pages rather than requiring a contiguous block per request, brings memory waste below 4%, which allows a larger batch at no latency cost under constant load. RadixAttention, at SGLang, reuses already-computed prefixes, and its authors state it outright: the technique “benefits both throughput and latency … also reduces the computation of prefill, thus decreasing the first token latency”. FlashAttention A tile-by-tile implementation of attention that never materializes the full attention matrix in HBM. It cuts memory consumption sharply and speeds up the computation, most of all on long contexts. Three versions (v1/v2/v3), each tuned for a GPU generation. , fused kernels and CUDA graphs cut the fixed cost of every iteration, which helps everyone.
These optimizations have something in common: they push the limit outward. They do not slide a point along a curve, they deform the curve itself. As long as one of these is still left to enable, talking about a trade-off makes no sense: you simply have work outstanding.
The trade-off does exist, and it appears once that work is done. At fixed hardware, model and software stack, with all these techniques enabled, an arbitration remains along the limit. The scheduler does not choose the limit. It chooses where on it you sit.
Which leaves the question of between what and what. And it is not between throughput and latency.
What a prefill token does inside a decoding batch
Go down into the loop, because everything else follows from it.
A modern inference engine does not reason in requests but in a per-iteration token budget. On each forward pass it has an envelope, say 2,048 tokens, and it fills it with whatever is at hand. A request in the middle of decode The autoregressive generation phase of an LLM: one token is produced at a time while rereading weights and the relevant context state. At low batch sizes, that traffic can make decode memory-bandwidth-bound. Its share of total cost still depends on input and output lengths, batching and cache reuse. consumes exactly one token: the one it is about to produce. A request that has just arrived demands its entire prompt at once, which can be several thousand tokens. Chunked prefill The opening phase of LLM inference: prompt tokens are processed in parallel to build the context state. That reuse raises arithmetic intensity and can make the phase compute-bound; the exact regime depends on the model, batch, context and backend. exists precisely for this: rather than making everyone wait while an 8,000-token prompt is swallowed, you slice it and slot the pieces into successive iterations.
Here is what happens then, and it is the heart of the matter. Every prefill token you insert takes a place in the envelope and lengthens the iteration. And every decoding request is waiting for that iteration to end before receiving its next token. One incoming request therefore makes every conversation already under way wait. Time between tokens degrades, for everybody, because of a prompt that belongs to nobody but the latest arrival.
Refuse to insert those tokens and you get the opposite symptom. Conversations under way run smoothly, their cadence is steady, but the newcomer waits its turn in the queue. Its time to first token stretches.
That is the real arbitration. On one side the initial wait of whoever arrives, on the other the regularity of the stream for those already served. Two latencies, not a throughput against a latency. And they oppose one another because they compete for the same envelope.
vLLM’s documentation gives the direction of the effect without hedging: a smaller budget value “achieves better ITL”, a larger one “achieves better time to first token”. The same paragraph recommends going past 8,192 to chase throughput. Three objectives, one parameter, and the choice is yours.
The clearest proof comes from the paper that introduced the technique. Sarathi-Serve, presented at OSDI in 2024, decomposes the effect of its two ingredients and the result is counter-intuitive: chunked prefill taken alone increases time to first token, because processing a prompt in pieces is slightly less efficient than swallowing it whole. Hybrid batching taken alone increases time between tokens. You need to combine the two to get a good compromise. Each lever on its own degrades one of the two latencies.
Take care not to confuse that number with the per-iteration token budget we have been discussing throughout. One counts conversations decoding in parallel, the other an envelope of work per pass. Both are expressed in tokens because a decoding request produces exactly one per iteration, but they are distinct quantities, and a budget of 2,048 in no way means you are seven times above the threshold.
Sarathi observes something of the same family on an entirely different configuration, without it being the same measurement: on a Llama-13B served by an A6000, raising the prefill token count stops delivering throughput past roughly 512 tokens. Two pieces of hardware, two quantities, a corroboration of order rather than a confirmation: there is a point past which filling further stops paying.
So keep the full statement, because the short version is wrong: above the critical batch, every prefill token inserted into a decoding batch improves your throughput and degrades the time between tokens for every other user in the batch. Below it, it does almost nothing.
The price on the label
Which leaves what it costs to move along that limit. The public numbers on the subject are plentiful and nearly all unusable, because a latency without its model, its hardware, its input and output lengths and its concurrency level means nothing. One table escapes that flaw.
| Concurrency | Time to first token | Average latency | Throughput |
|---|---|---|---|
| 1 | 48 ms | 136 ms | 67 tok/s |
| 25 | 657 ms | 958 ms | 217 tok/s |
| 100 | 2,051 ms | 3,562 ms | 250 tok/s |
| 500 | 5,653 ms | 6,791 ms | 614 tok/s |
Look at the two factors separately, because that is where the article turns. Between the first row and the last, throughput was multiplied by 9. Time to first token was multiplied by 117. You are not buying throughput at the price of a proportional latency degradation: the degradation runs far faster than the gain.
Two caveats on this table. NVIDIA raises the first: it is generated automatically, and the very high waits at the top of the curve also contain client-side queueing, not only server compute. The second matters more for our subject. With only ten output tokens, this table says almost nothing about the time between tokens. It lights one side of the arbitration perfectly, the initial wait, and leaves the other in the dark. We found no public table documenting both with a complete configuration.
What chunked prefill delivers against that is measured in serving capacity at a held SLO Service Level Objective. A quality target a system sets for itself, say a time to first token under 500 ms at the 99th percentile. Distinct from an SLA, which is the contractual commitment you can enforce: in LLM inference, SLOs are everywhere and latency SLAs almost nowhere. . Sarathi-Serve reports 2.6 times the capacity for a Mistral-7B on one A100, 3.7 times for a Yi-34B on two A100s, and up to 5.6 times for a Falcon-180B on eight A100s with pipeline parallelism. Those numbers do not transport elsewhere, but their order of magnitude says the essential: tuning this slider properly does not return a few percent.
What the three teams actually decided
Now revisit the three defaults from the opening, with the mechanism in mind.
| vLLM | SGLang | TensorRT-LLM | |
|---|---|---|---|
| Batch order | Decode first | Prefill first | Decode first |
| Chunked prefill | On, 2,048 budget | On, 8,192 chunks | Off |
| Memory saturation | Preempt by recompute | Retract then recompute | No preemption |
| What is protected | Inter-token cadence | Throughput | Predictability |
| Priority scheduling | Available, not default | Available, not default | Available, not default |
vLLM treats prompt and generated tokens uniformly in a single scheduler, with a fixed budget it distributes dynamically across requests. It serves already-running requests first, then admits new ones if the budget allows. Its 2,048-token budget therefore leaves little room for prefill on each iteration, which protects the cadence of conversations under way. When memory saturates it preempts by recompute rather than by swapping to host memory, on the grounds that recomputing costs less than moving in this architecture.
SGLang does the opposite in two places. It forms a prefill batch as soon as it can, before falling back to decode, and its 8,192-token chunks fill the envelope generously. It is also the only one of the three to reorder its queue by cache: its default policy picks the longest already-computed prefix, which raises the reuse rate at the cost of scheduling overhead. In vLLM, a cached prefix reduces the cost of a request once admitted but does not change its place in the queue.
TensorRT-LLM, finally, prefers not to promise what it cannot hold. By reserving at admission the memory needed for the complete sequence, it guarantees no accepted request will be interrupted. Its documentation presents the alternative plainly: “If the goal is to maximize throughput, MAX_UTILIZATION should be tried, keeping in mind that it may impact latency if requests have to be paused.” Maximum throughput exists, it is one flag away, and it is not the default.
A point of vocabulary is due here, because it separates those who read the documentation from those who read blog posts. What vLLM calls continuous batching, the NVIDIA ecosystem calls in-flight batching. The term “dynamic batching”, at Triton, means something else: grouping waiting requests at the request level and then running the batch to completion, which is static batching with dynamic batch formation. Using one for the other gets noticed immediately.
One last thing common to all three, and it counts: priority scheduling exists everywhere and is the default nowhere. All three start from first come, first served, with decode priority for two of them. If you have tenants or traffic classes to treat differently, that is an explicit setting to enable, not a behaviour you get.
The defaults do not know your service objective
It would be easy, at this point, to conclude that these defaults are badly set. That would be unfair and wrong. Each is a general-purpose compromise, documented, chosen by people who knew what they were sacrificing. What none of them can know is what you serve.
Those percentages do not read as gains wrung out of an already-tuned system. They are measured against the shipped default configuration, applied to a workload it was never designed for. These are not heroic optimizations, they are defaults collapsing on those particular workloads. Which is exactly the argument.
Which leaves the question of what value to push towards, and we will not give a range. That is deliberate. The ones circulating in vendor guides do not agree with each other on the direction of the effect: some present large budgets as protecting the initial wait, others as protecting cadence, and both readings turn up in neighbouring documents. Publishing one of those brackets would send you to tune the wrong end.
What is solid is the direction, and the engines themselves document it. A smaller budget protects the cadence of those already being served and makes the arrival wait. A larger budget does the reverse. Start from your engine’s default, measure your two latencies separately under your real workload, and move the slider towards whichever one is violating your objective. The disagreement between the guides is, incidentally, the best possible argument for measuring it yourself.
Three layers, not one
An article that stopped there would be dated, because part of the arbitration has left the engine.
Prefill/decode disaggregation, which we followed as it came off the page, does not settle the trade-off described above: it removes it. By placing the two phases on separate GPU pools, no prefill token ever slips into a decoding batch, and the interference the whole mechanism rests on disappears. DistServe reports 7.4 times more requests served, or a 12.6 times tighter SLO, holding latency for more than 90% of requests. Splitwise, at Microsoft, measures 1.4 times the throughput at 20% lower cost, or 2.35 times the throughput at constant power and cost budget.
A third layer sits above, that of routing between replicas. Dynamo’s router picks its target by combining already-cached prefix overlap with each instance’s load, using an adjustable weight that arbitrates, again, between time to first token and time between tokens. The same trade-off, one storey up.
Should we conclude the engine scheduler has become marginal? Dynamo’s documentation answers for itself, and its candour deserves quoting: “For many standard chatbot or moderate RAG workloads, aggregated serving is still the simpler and often better default.” Disaggregation is paid for in cache transfer between stages, and a third-party measurement estimates that transfer can erode 30 to 50% of the theoretical gain when the network fabric is not up to it.
So here is where to place this article’s own slider. On a fleet of dozens of nodes handling long contexts, the arbitration lives in the router and in the separation of stages. On a co-located deployment, which is most of what runs today, it lives in your engine’s token budget. And do not confuse that scheduler with the operating system’s: one decides which token to compute on the next pass, the other which thread gets a core. Both matter, they do not solve the same problem.
Nobody guarantees your latency
Let us end with what should make this parameter matter to you.
We looked at what inference providers commit to contractually. Azure guarantees 99.9% monthly availability and explicitly excludes latency from scope. AWS Bedrock commits on availability alone. Baseten writes 99.9% system availability in plain terms, with latency absent from the contract and reduced throughput explicitly excluded from credit grounds. Vertex AI advertises 99.5% for online Gemini inference. Together shows 99% availability. Groq, whose entire positioning rests on speed, promises only “commercially reasonable efforts” in its public terms. Anthropic publishes no service agreement.
One public exception exists, and it is instructive. OpenAI’s Scale Tier carries a quantified throughput commitment: 99% of requests above 25 tokens per second, measured as average per-request latency on a per-minute basis, aggregated over the month. The standard tier has no latency commitment at all.
Azure and AWS do sell predictable latency, through provisioned throughput units billed hourly. But read the documents: that reserved capacity lives outside the service agreement. You are buying a probability, not an enforceable clause.
What this means for you is simple. Your latency commitment is held by nobody on your behalf. It is held by your engine’s token budget, by how your router spreads requests, and by your decision to separate the two phases or not. Those are three settings nobody took for you, and the first of the three is an integer.
The question that comes next is measurement. None of the figures cited here transports onto your workload, and two benchmark harnesses do not compute inter-token latency the same way: one excludes the first token from the calculation, the other includes it. Before tuning anything, you need to know what you are measuring. That is a subject of its own, and we will come back to it.
Sources and method
Default values cited reflect the state of July 2026 and change from version to version: check yours before acting. No latency figure is reproduced without its model, its hardware, its input and output lengths and its concurrency level.
Verified facts
Scheduler behaviour. vLLM optimization documentation for chunked prefill being on by default in V1, the max_num_batched_tokens budget, the direction of its effect on ITL and time to first token, and preemption by recompute. vLLM V1 release post of January 27, 2025 for the unified scheduler design. SGLang documentation and GitHub discussion #1163 of August 25, 2024 for the 8,192 chunk size, the 16,384 prefill budget, prefill-decode fusion being off by default and the cache-aware queue policies. TensorRT-LLM documentation for chunking being off by default, the GUARANTEED_NO_EVICT policy and its MAX_UTILIZATION alternative.
Pushing the limit outward. Anyscale post of June 22, 2023 (Cade Daniel, Chen Shen, Eric Liang, Richard Liaw) for continuous batching and the simultaneous improvement in throughput and median latency. The vLLM paper (PagedAttention, SOSP 2023) for memory waste brought below 4%. The SGLang paper (arXiv 2312.07104) for RadixAttention’s effect on throughput and on first-token latency.
Mechanism and regime. Sarathi-Serve (OSDI 2024, arXiv 2403.02310) for the decomposition of the two levers, the serving capacity gains (2.6x for Mistral-7B on one A100, 3.7x for Yi-34B on two A100s, up to 5.6x for Falcon-180B on eight A100s) and the diminishing returns past roughly 512 prefill tokens on Llama-13B and A6000. Austin et al., How to Scale Your Model (Google DeepMind, 2025, chapter 7) for the critical batch derivation.
Throughput-latency curve. NVIDIA NIM performance table for the Nemotron Safety Guard Multilingual 8B on A100 80 GB SXM in BF16, 500-token inputs and 10-token outputs, the only source found publishing model, hardware, lengths, concurrency, time to first token and throughput together.
Tuning against the defaults. SCOOT (arXiv 2408.04323, section 5.2) for the reductions in time to first token and time per output token obtained by Bayesian optimization on two real applications.
Upper layers. DistServe (OSDI 2024, arXiv 2401.09670) and Splitwise (ISCA 2024, arXiv 2311.18677) for disaggregation gains. NVIDIA Dynamo documentation for the cache-aware router, the prefix overlap weight and the recommendation in favour of aggregated serving on common workloads.
Service commitments. Public contractual pages from OpenAI (Scale Tier), Microsoft Azure OpenAI, AWS Bedrock, Baseten, Google Vertex AI, Together and Groq, together with the absence of a public service agreement at Anthropic.
Credible estimates
The threshold of roughly 295 concurrently decoding requests in BF16 on H100 is an analytical derivation from the ratio between compute throughput and memory bandwidth, not a measurement on a real workload. It gives an order of magnitude for the crossover, not a threshold to program. The 512-prefill-token ceiling Sarathi observes on another configuration is a corroboration of order, not a confirmation: the two measurements share neither quantity nor hardware.
We deliberately publish no token budget range. The vendor recommendations we gathered contradict one another on the direction of the effect, some associating large budgets with a better initial wait and others with better cadence. Only the direction documented by the engines themselves is reproduced here.
The erosion of 30 to 50% of the disaggregation gain by cache transfer comes from a third-party measurement and depends entirely on the network fabric available.
What we could not establish
The real default for max_num_batched_tokens in online serving is not settled by a single official source: legacy documentation gives 2,048, current recommendations and AMD’s documentation give 8,192, and the value may be computed dynamically from available memory. We quote the direction of the effect and the range rather than a value.
No official measured preemption cost is published by any of the three engines. The figures in circulation come from third-party papers or contribution threads.
The line-by-line description of the scheduling loops comes from sources derived from the code rather than a direct reading at the current version. We stay with the behaviour the projects themselves document.
The gains vendors announce on their own platforms are measured on optimized configurations and are not a guarantee of production behaviour.
