The GPU disappears behind the API
Comparing two GPUs is a bounded problem. Fix the model, precision and batch, then put the timer around compute. Comparing two inference services adds everything that method excludes: router, queue, scheduler, cache, network and replica count behind the HTTP address. That whole system is what you buy when your application calls an API.
MLPerf Endpoints moves the instrument accordingly. The load generator sits in front of the endpoint, where an application would, and sends requests through three profiles. Offline presents all work together to find maximum throughput. Online spaces requests through a Poisson process, reproducing independent arrivals around a target rate. The third profile holds a fixed number of requests in flight: when one response completes, another takes its place.
The last profile defines the v0.7 publication: all 100 performance-point files use load_pattern: concurrency. It does not ask “how much can this server produce?” It asks “what happens at 32, 512 or 8,192 simultaneous requests?” At each level, the harness records completed requests per second, time to the first useful chunk, average time per output token and end-to-end latency. The endpoint is genuinely the measurement boundary. It is not necessarily a cloud product that anyone can buy: several v0.7 submissions use internal addresses and describe a complete on-premises system.
A curve, not a podium
NVIDIA’s GPT-OSS-120B results expose the central mechanism. The declared system remains one GB300 NVL72 rack, the model remains MXFP4 and the performance dataset remains fixed. Concurrency and server configuration are adjusted together to find several frontier points. At 544 simultaneous requests, the system completes 101.8 requests per second. At 22,528, it completes 775.6. Throughput rises by 7.6×, but individual requests pay for that density.
| Concurrency | Requests/s | TTFT p99 | TPOT p99 | Total latency p99 |
|---|---|---|---|---|
| 544 | 101.8 | 893 ms | 4.24 ms/token | 8.21 s |
| 22,528 | 775.6 | 15.72 s | 23.34 ms/token | 46.51 s |
At 22,528 requests in flight, every rack-second produces many more answers. Yet 1% of requests wait longer than 15.72 seconds for the first useful chunk, and 1% take more than 46.51 seconds to finish. Both points are exact for the published workload. Neither is “the performance” of the system by itself.
MLCommons’ public visualization emphasizes TTFT p95. The table deliberately uses p99 from the same raw files to inspect a harsher tail. Its values should not be compared directly with the p95 axis on the Endpoints website.
These are not two runs where one knob changed. Between their server files, maximum decoder batch grows from 32 to 512, context-server batch from 4 to 16, context token budget from 16,384 to 65,536 and attention data parallelism is enabled. CUDA graphs capture different batches too. The benchmark therefore shows two optimized configurations on the same declared system, not the sensitivity of one fixed configuration to concurrency. That is consistent with searching for a Pareto frontier, but it prevents attributing the gap to request count alone.
The public package cannot reconstruct the second run’s topology with certainty. For point 544, the result name and source files describe four context servers, seventeen generation servers and one front end. For point 22,528, the name archived in results/config.yaml encodes ng48_nc24_fe48, while the client file and gptoss_config.yaml describe twenty-eight context servers, eleven generation servers and eleven front ends. run_metadata.json confirms 72 GPUs, a disaggregated stack and batch size 512, but does not resolve the contradiction. The measurements remain verifiable; assigning them to one exact topology does not.
This curve is a SLO Service Level Objective. A quality target a system sets for itself, say a time to first token under 500 ms at the 99th percentile. Distinct from an SLA, which is the contractual commitment you can enforce: in LLM inference, SLOs are everywhere and latency SLAs almost nowhere. . If your product requires a first useful chunk below one second at p99, the second point does not exist for your decision, regardless of its throughput. We use admissible throughput for the QPS of a point retained only after two separate checks: its latency and error distributions satisfy the SLO, then its configuration passes the quality threshold in the separate evaluation. This is an analytical term defined here, not an MLPerf Endpoints 0.7 metric.
The distinction prevents a misleading ranking. One provider can increase concurrency until its queue fills and publish many requests per second. Another can stop earlier to protect the first token. Ranking only by throughput rewards the willingness to degrade experience, not the ability to serve the same constraint.
TTFT, TPOT and QPS describe different waits
TTFT measures the interval between sending the request and receiving the first content or reasoning chunk that the client treats as useful. It includes HTTP transit, queueing, prefill and the start of decoding. It is the visible pause before an interface begins responding. In MLPerf Endpoints, it also includes transit through the measurement client’s internal transport. The client is designed to reach tens of thousands of requests per second, but its cost never becomes mathematically zero.
The name hides a decisive nuance: the client timestamps the first useful chunk received, not the first token computed inside the server. NVIDIA sets stream_interval to 100 on both stages. In TensorRT-LLM, this option groups responses every 100 iterations but explicitly exempts the first token to preserve TTFT. It does not delay the first chunk by 100 tokens. It does make later cadence coarse, preventing this run from revealing pauses between individual tokens.
TPOT divides the interval from the first chunk to completion by the number of counted tokens after that chunk. It is an average across the answer, not the exact interval between every token pair. Grouping by 100 approximately preserves that average over a long output but hides regularity. An endpoint can report good TPOT while emitting bursts separated by pauses. Our scheduler analysis explains the mechanism that trades this time against the first token; the question here is whether two providers computed both quantities the same way. The common client fixes its method, but server streaming interval remains a parameter to inspect.
QPS counts requests, not work. A request producing 128 tokens and one producing 3,700 each count as one. The first release’s three models have different profiles by design: Llama 3.1 8B uses a short summarization workload, while DeepSeek-R1 emits much longer reasoning. Dividing their QPS values does not compare tokens per second or useful capacity.
Quality closes the shortcuts
A serving optimization can gain throughput by lowering weight precision, truncating outputs or changing the model behind a commercial name. If answers degrade, the benchmark no longer accelerates the same service. MLPerf Endpoints therefore pairs performance measurement with an evaluation phase.
For GPT-OSS-120B, NVIDIA’s submission runs AIME 2025 over 30 problems repeated eight times, GPQA over 198 questions repeated five times and LiveCodeBench over 1,055 tasks repeated three times, at temperature 1. Point 544 scores 80.42%, 73.74% and 84.61%. Point 22,528 scores 78.75%, 73.84% and 85.56%. The variation confirms that evaluation is stochastic and specific to each run. Because the policy repository linked by MLCommons is unavailable, these files alone cannot prove which official threshold each MXFP4 transformation had to pass. They also do not predict quality in French, JSON output, retrieval, or tool use.
The first cycle covers only three model families: Llama 3.1 8B, DeepSeek-R1 and GPT-OSS-120B. MLCommons announces agentic workloads for version 1.0. The current coverage is enough to verify that the protocol connects performance with quality. It is not enough to represent a provider’s catalog or the applications that call it.
“Quality” also has two meanings here. The benchmark measures answer correctness on its datasets. A buyer also expects quality of service: HTTP error rate, overload rejection, availability, regional variation and failure behavior. Version 0.7 publishes failed requests in each run, but it does not conduct a multi-day availability campaign. An error-free run is not a production commitment.
Cost is still missing from the equation
MLCommons states four principles: currency, coverage, comparability and context. Its comparability definition targets results normalized by cost or power. The announcement promises more buyer-oriented rules and “normalization” for MLPerf Endpoints 1.0, without publicly specifying whether the first formula will address price, power or both. Version 0.7 supplies none of those calculations.
Public files describe hardware, model, precision, runtime and results. They contain no normalized hourly rate, price per million tokens or measured power. An internal NVIDIA rack, a CoreWeave endpoint and a Dell server cannot be reduced to one bill from these data alone. Version 0.7 measures capacity. It does not calculate what that capacity costs.
The purchasing calculation is still simple to state for a resource billed by the hour:
For a token-priced API, catalog input and output rates replace hourly price, but the principle survives. Apply them to observed sequence lengths, include retries after errors, then reject points outside the service objective. MLPerf Endpoints could provide the missing term: measured throughput under constraint, provided that price, model and region are frozen at the same instant.
What v0.7 can establish
The first release supports three solid decisions. It reveals a system’s curve as concurrency rises. It compares configurations of the same model under a common load generator. It publishes enough metadata and raw results for a reader to verify scope before interpreting a point.
It cannot identify “the fastest provider” without naming a model and latency limit. It cannot identify “the cheapest” because price and energy are missing. It measures neither cold starts, commercial rate limits, regional outages nor daily variation on a shared endpoint. Participant submissions are also not measurements from a laboratory that anonymously purchased every offer; organizations submit their own systems and configurations under the protocol.
One documentary weakness remained open at this article’s cutoff. The public results repository links to mlcommons/endpoints_policies, but the web address returned 404 and git ls-remote reported no repository on August 30, 2026. The harness code, configurations and results are public; the exact v0.7 policies are not accessible from their official link. The published package alone therefore cannot verify all validity and quality thresholds.
| Question | v0.7 answer | Still required |
|---|---|---|
| How much load? | QPS and concurrency | Real hourly traffic profile |
| How much waiting? | TTFT, TPOT, total latency | Product-specific SLO |
| How accurate? | Model-specific datasets | Internal domain evaluation |
| How many errors? | Failures during the run | Availability over time |
| What cost? | Not normalized | Price, energy and admissible throughput |
Conclusion
MLPerf Endpoints 0.7 cannot yet answer “which provider should I buy?” It corrects a deeper error: the idea that an inference service can be summarized by one number. The real object is a curve connecting load, first token, generation cadence and quality, from which each buyer selects a different region.
Version 1.0 will matter if its normalization connects a measured resource, a cost or power figure and an admissible latency point. Dividing a catalog rate by maximum QPS would recreate the misleading ranking that the protocol has begun to avoid. The denominator must be stricter: a configuration that preserves quality and completes the workload within the promised time. Only then does a performance benchmark become a purchasing instrument.
Sources and method
Editorial cutoff: August 30, 2026. Every measured figure is a verified fact from MLCommons primary sources and submission files. Cost formulas are explicit derivations; no provider price is injected into the v0.7 results.
Announcement and scope. MLCommons, MLPerf Endpoints v0.7: A Foundation Release, July 28, 2026, identifies the five initial participants, three benchmarks and four project principles. The official benchmark page documents the visualization’s TTFT p95, operating points, availability status and result review. The announcement promises normalization, buyer-oriented rules, agentic workloads and continuous submissions for version 1.0; only the comparability principle says “cost or power,” without committing to both metrics in 1.0.
Measurement harness. The official mlcommons/endpoints repository defines the three load profiles, client architecture, QPS, latency, TTFT and TPOT metrics, and the AIME, GPQA and LiveCodeBench evaluations. The current TTFT and TPOT definitions were checked in the code and client documentation. TensorRT-LLM’s official documentation defines stream_interval and its first-token exemption. The MLCommons repository still marks itself “Alpha” in pyproject.toml at the cutoff. The v0.7 summaries carry short hash 629c1bc, which the public history did not resolve on August 30, so the current code is not presented as a bit-exact revision of the publication harness.
Raw results. The official MLPerf Endpoints v0.7 results repository is pinned at revision b5328511abb79c84b520e3c7775fb2336b773f37. All 100 performance-point files were checked and use concurrency load. Table 1 uses point_544 and point_22528. Requests per second are recalculated as n_samples_completed ÷ (duration_ns / 10⁹); other figures are the published 99th percentiles converted from nanoseconds. The point 544 server configuration and point 22,528 configuration establish changes to batching, attention parallelism and CUDA graphs. Their descriptions of point 22,528 topology conflict with the archived result name; this article retains that caveat.
Model, system and quality. NVIDIA’s GPT-OSS-120B system description supports the GB300 NVL72 rack, MXFP4 precision and declared maximum concurrency. Accuracy files for point 544 and point 22,528 provide the sizes, repetitions and separate AIME 2025, GPQA and LiveCodeBench scores.
Audited limitations. The official roadmap, issue #223, tracks harness stability, reliability, coverage and parity. The v0.7 results README links to https://github.com/mlcommons/endpoints_policies, which returned 404 at the cutoff. This is reported as a dated documentary state, not proof that the rules exist nowhere.
Claims excluded. Results are not converted into a global ranking, cost per token or cost per request. No vendor number is transferred to another model, region or output length. Admissible throughput is an analytical definition proposed to connect performance and purchasing; it is not a metric published by MLPerf Endpoints 0.7.
