MLPerf RAG finally measures the full pipeline, but not yet the service

A RAG system is not one model call

A conventional LLM benchmark receives a prompt, runs one model and counts tokens. That unit works while the compute path stays inside the model. Retrieval-augmented generation adds a pipeline before the answer: encode the question, query an index, rerank passages, discard those that do not address the need, decide whether the evidence is sufficient, then reformulate the search when a link is missing.

Each stage stresses hardware differently. The embedder processes many short passages. The index traverses an in-memory structure. The reranker compares query and document tokens. Generative models alternate prefill prefill The opening phase of LLM inference: prompt tokens are processed in parallel to build the context state. That reuse raises arithmetic intensity and can make the phase compute-bound; the exact regime depends on the model, batch, context and backend. , decode decode The autoregressive generation phase of an LLM: one token is produced at a time while rereading weights and the relevant context state. At low batch sizes, that traffic can make decode memory-bandwidth-bound. Its share of total cost still depends on input and output lengths, batching and cache reuse. and short classification calls. Timing only the final generator is like profiling a function while ignoring the calls, copies and queues around it.

MLCommons addresses part of that gap with MLPerf End-to-End RAG, added to Inference v6.1 and introduced on August 26, 2026. For the first time in MLPerf Inference, a multi-component workload follows a question through retrieval and reasoning instead of isolating a model. The name is defensible, provided that we inspect where the timer starts, where it stops and what the protocol fixes in between.

“End-to-end” is split across two benchmarks

The corpus does not pass through indexing and then all 824 questions within one run. MLPerf defines two independent workloads.

E2E-RAG-DB begins with 2,515 English Wikipedia HTML files. It extracts the page body, preserves tables and lists as text, chunks the result, computes embeddings and builds a FAISS HNSW index. The score is documents per second. Model loading is outside the timer.

E2E-RAG-QnA loads that prebuilt database and processes 824 questions. The score is tasks per second, where one task is one question carried through its final answer. This avoids a misleading tokens-per-second aggregate: one task invokes two GPT-OSS sizes, an embedder and a reranker, with a variable number of rounds.

Workload	Input	Timed work	Metric	Outside the timer
E2E-RAG-DB	2,515 HTML pages	Parsing, chunking, embedding, HNSW index	Documents/s	Model loading
E2E-RAG-QnA	824 questions + built database	Retrieval, reranking and LLM calls	Tasks/s	Model loading, final judge

“End-to-end” therefore describes each measured path, not the entire life cycle of a document database. A system can lead ingestion and lose QnA, or the reverse. Adding the scores would be meaningless: one counts documents, the other questions, and an index can serve an arbitrary number of requests after construction.

Ingestion turns 2,515 pages into 107,000 retrieval units

The corpus is a fixed Wikipedia snapshot distributed with the benchmark. Freezing it prevents an edited page from moving passages and changing retrieval between submissions. The 2,515 pages become about 107,000 passages of 768 characters, with a 32-character overlap. Each passage retains the source page URL.

Overlap protects a narrow boundary region: a date or name cut at the end of one segment appears again at the beginning of the next. It does not preserve structure. A flattened table cell becomes a text sequence; a relationship encoded in layout can disappear while every character remains. The ingestion score rewards the speed of this defined path, not the best available way to index a PDF collection, SQL database or multilingual corpus.

e5-base-v2 maps every passage to a 768-dimensional vector. FAISS then arranges them in an HNSW graph. The first release fixes M = 32, efConstruction = 200 and efSearch = 100. That constraint has a purpose: MLCommons does not yet have an official retrieval-quality threshold that would let two different indexes prove equivalence. Fixing the algorithm reduces freedom, but prevents a throughput score from being gained by silently sacrificing recall.

One question can trigger five rounds and more than twelve LLM calls

FRAMES contains 824 questions whose answers require information from several pages. A question does not hand the pipeline the right search terms. The system may need to identify one entity, use it to find the next, then determine whether the chain is complete.

Rewrite up to 3 subqueries GPT-OSS-120B

Retrieve embedding + HNSW e5-base-v2

Rerank top candidates per subquery ColBERTv2

Filter relevant passages GPT-OSS-20B

Sufficiency evidence complete? GPT-OSS-120B

Answer retained facts GPT-OSS-120B

The rewriter produces up to three subqueries. e5-base-v2 encodes each one and searches the index. ColBERTv2 reranks the candidates for each subquery through late interaction: token-level representations survive instead of collapsing the entire passage into one vector. The controller then removes URLs it has already seen. GPT-OSS-20B assigns binary relevance to each newly retrieved document.

GPT-OSS-120B examines the retained set. If one relation is missing, it rewrites and starts another cycle. The protocol permits up to five rounds and requires that ceiling for every submission. When evidence is judged sufficient, or after the fifth round, the same GPT-OSS-120B produces the final answer. The reference implementation can therefore make twelve generative calls or more for a single task.

Capacity planning no longer follows the parameter count on one model card. GPT-OSS-120B is a MoE MoE Mixture-of-Experts. An architecture where the network is split into many experts, of which a router activates only a small subset per token. Per-token compute follows the number of active parameters; memory follows the total count, since every expert must stay resident in VRAM, ready to be called. with 117 billion parameters and 5.1 billion active per token; GPT-OSS-20B has 21 billion total and 3.6 billion active. The larger model fills three roles with a variable call count, while the 20B absorbs higher-volume document filtering. e5-base-v2 and ColBERTv2 each contain only 110 million parameters, but their work repeats: e5 encodes every passage during DB and each subquery during QnA, while ColBERT reranks every set retained by retrieval.

Component	Model	Published size	Reference precision	Place in measurement
Embedding	e5-base-v2	110M	FP32	DB + QnA
Reranking	ColBERTv2	110M	FP32	QnA
Document grader	GPT-OSS-20B	21B / 3.6B active	MXFP4	QnA
Rewrite, sufficiency, answer	GPT-OSS-120B	117B / 5.1B active	MXFP4	QnA
Accuracy judge	Llama 3.1 8B	8B	BF16	After the run

Scheduling is the real benchmark

In a single-model workload, increasing batch size fills the GPU more effectively until memory or a latency constraint intervenes. Here the batch crosses a chain whose stages have different service rates. Ten tasks may create thirty subqueries, one hundred passages to rerank, dozens of document decisions and generations of different lengths. Accelerating one stage moves the queue to the next.

A submitter can place the embedder on CPUs and the generative models on GPUs, isolate the two GPT-OSS models on separate accelerators, share a device through memory partitioning or select different precisions while staying above the quality threshold. Tasks can overlap too: GPT-OSS-120B rewrites a query for batch A while ColBERT handles batch B and the 20B grades documents for batch C. The stack behaves like a processor pipeline. Throughput depends less on the fastest stage than on preventing bubbles between them.

Offline makes that scheduling freedom more important. LoadGen presents all 824 tasks as one batch, allowing the system to group and order them for throughput. The prefix that grows across rounds also creates opportunities for KV cache KV cache The stored key and value vectors an LLM has already computed for every token it has processed. It avoids recomputing attention over the whole history, at the cost of a memory footprint that grows with the context length. reuse and prefix caching within a task. The rules forbid reusing content-derived results across different questions.

Tasks per second is therefore the right aggregate for this pipeline: it absorbs every stage and its imbalance. It says nothing about the latency experienced by one request. A system may report excellent throughput by keeping every accelerator full while a question waits behind a large batch.

The performance test replays decisions, not behavior

A multi-hop loop creates a reproducibility problem. GPT-OSS-120B samples at temperature 1 with top_p = 1. A difference in the first rewrite can change retrieved passages, round count and every later call. Two machines would no longer receive the same amount of work.

MLPerf therefore separates accuracy from performance. The accuracy run uses live model outputs. The performance run still calls every LLM and times its inference, but substitutes a recorded answer before the pipeline continues. Every system receives the same subqueries, passages and sufficiency decisions at the next stage.

The mechanism resembles replaying a branch trace in a profiler: the instruction executes, but control flow comes from a shared reference. It compares the cost of compute and orchestration without letting randomness change the workload volume.

The distinction blocks two readings. A high QnA score does not prove that the system writes better queries, because their downstream effects are substituted. It also does not prove that a different model preserves its own control dynamics. Quality is checked in a separate run.

The accuracy threshold protects a weak reference

MLCommons reports 35% final-answer accuracy across 824 questions, with an observed ±3 percentage-point variation from GPT-OSS nondeterminism. The rules give 68% oracle accuracy when the correct documents are supplied, then attribute the gap to approximate retrieval, reranking, context windows and multi-hop reasoning. Iteration raised the reference from 20% to about 35%.

A submission is valid when it preserves at least 97% of reference accuracy. Applied to the rounded 35%, that places the threshold near 34%. The constraint prevents quantization or optimization from trading too many correct answers for throughput. It does not turn 35% into a product target: the judge still rejects nearly two out of three final answers on this workload.

The reasoning breakdown reinforces the limit. MLCommons reports 38% for multiple-constraint questions, 34% for post-processing, 32% for temporal reasoning and 31% for tabular and numerical questions. These numbers describe the reference stack, not a universal property of RAG. Character-based chunking preserves tables poorly, and no code interpreter handles calculation.

Retrieval quality has diagnostic reference values of 75% precision, 70% recall and 69% F1, but it is not an official metric. A separate integrity check enforces the index parameters and requires at least 95% mean overlap between local top-k URLs and the reference manifest on probe queries. That threshold measures parity with the reference database, not relevance to the correct pages. QnA compliance automatically checks only the final generator’s average output length, 273.81 tokens within ±10%. Retrieval, reranking and intermediate calls require reviewers to inspect the detailed log.

What the first ranking will not establish

The Server scenario is reserved for later cycles. The initial release imposes no gradual request arrivals, queueing-latency target, TTFT or latency percentile. It tests neither admission control, cancellation nor tenant priority. Its score answers “how many tasks can this stack finish when all work is visible?” rather than “how long does my user wait?”

The corpus also bounds extrapolation. It contains 2,515 English Wikipedia pages selected to answer FRAMES questions. No scanned PDF, access-control rule, duplicate document, incremental update or unanswerable request stresses the chain. Ingestion measures a complete rebuild, not the freshness of a continuously updated index.

No official E2E-RAG hardware result was public at this article’s August 30, 2026 cutoff. The announcement defines a workload and its references, not a winner. A headline already claiming that a GPU or vendor “wins MLPerf RAG” would confuse the protocol with submissions that had not yet been released.

How to read future results

The DB score should remain attached to ingestion time and embedder configuration. It matters to a team that regularly rebuilds large indexes; it carries little weight when a database is built once and queried for months.

The QnA score should be read with the MLPerf division, achieved accuracy, accelerator count and type, power and placement of timed components. Two systems at the same throughput can embody opposite architectures: one may replicate GPT-OSS-20B to clear the document queue, while another reserves more GPUs for 120B reasoning rounds. The system score intentionally hides those choices; the submission description must expose them again.

An acquisition decision still needs a test that sends realistic arrivals, enforces a latency percentile and reports cost per valid task. MLPerf E2E-RAG now provides a common workload on which to build that test. It does not yet provide the economic verdict.

Conclusion

The advance is not that MLPerf added another model to a list. It changed the object being measured: performance belongs to the system that places, feeds and synchronizes several models around an index, not to one neural network in isolation.

The limits of this first release already define the next step. When Server adds arrivals and latency, retrieval receives its own quality threshold and the path no longer depends on replayed decisions, the benchmark will move closer to a RAG service. Tools, structured indexes and autonomous decisions come after that. The problem will then be larger than timing a chain. It will be proving that an agent chose the right chain.

Sources and method

Editorial cutoff: August 30, 2026. Labels: verified fact for a citable primary source, estimate for an explicit calculation from sources and hypothesis for an interpretation without a measurement. No E2E-RAG performance result was public at the cutoff; this article analyzes the protocol and reference stack, not a hardware ranking.

Benchmark definition

• Verified fact. MLCommons, Introducing the MLPerf End-to-End RAG Inference Benchmark, August 26, 2026: DB and QnA workloads; 824 questions, 2,515 pages and about 107,000 passages; models, metrics, reference accuracy, Offline scenario, TEST09 and optimization directions.
• Verified fact. MLPerf Inference v6.1 documentation: E2E-RAG QnA and DB appear in the datacenter category with FRAMES as the dataset.

Pinned code and rules

• Verified fact. MLCommons reference repository at commit cfb0df14, reviewed August 30, 2026: DB/QnA scripts, MAX_ITERATIONS=5, MAX_SUB_QUERIES=3, TOP_K_RETRIEVER=10, model separation, performance-path replay, real LLM calls and the database integrity check with 95% top-k overlap.
• Verified fact. MLCommons E2E-RAG Inference rules at commit d3eba2f2: timed tasks, excluded model loading, 35% ±3 nondeterminism, 68% oracle accuracy, HNSW M=32, efConstruction=200, efSearch=100, cross-query cache prohibition and detailed-log review.
• Verified fact. The repository document Accuracy Tests vs Performance Tests confirms that performance mode executes LLM calls for timing, then returns recorded responses to the pipeline; accuracy mode uses live outputs.
• Document contradiction resolved. At the same commit, the TEST09 README retains an older 235.47-token mean. The applied audit.config sets 273.81 tokens and a 246.43 to 301.19 range, matching the MLCommons announcement. This article uses the concordant values.

Dataset and models

• Verified fact. Krishna et al., Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation, NAACL 2025: FRAMES construction, 824 multi-hop questions, reasoning categories and original measurements. The paper uses a different stack from the MLPerf reference, so its scores are not transferred to the 2026 benchmark.
• Verified fact. OpenAI, Introducing gpt-oss, August 5, 2025: GPT-OSS-120B contains 117 billion parameters with 5.1 billion active per token; GPT-OSS-20B contains 21 billion with 3.6 billion active; MoE weights are supplied in MXFP4.

Calculations and editorial limits

• Arithmetic estimate. “About 34%” comes from 35% × 97% = 33.95%. The 35% reference is rounded and varies by ±3 points, so this article does not present a hundredth-precision threshold.
• Dated observation. On August 30, 2026, MLCommons still listed Inference v6.0 as its latest results announcement, and no public inference_results_v6.1 repository existed in its GitHub organization. The absence of E2E-RAG hardware results is a cutoff-date observation, not a permanent property of the benchmark.
• Editorial interpretation. Calling this release a system-throughput benchmark rather than an interactive-service benchmark follows from Offline being its only scenario, the absence of a latency metric and decision replay in performance mode. This does not deny that the timed path is end-to-end; it defines that path’s boundary.