AI compute infrastructure: GPUs, runtimes, inference - LeCompute

Nobody knows how to measure coding agents anymore

redaction@lecompute.fr (Christophe Gerardin) — Mon, 13 Jul 2026 00:00:00 GMT

En bref

On July 8, 2026, OpenAI retracted its recommendation of SWE-Bench Pro (~30% broken tasks), five months after abandoning SWE-bench Verified, the instrument it had built itself. Three generations of benchmarks, three obsolescences coming ever faster.
The mechanism is metrological, not scandalous: models went from 23.3 to 80.3% in eight months and bump against a noise ceiling (~70%) made of defective tasks; beyond it, the gaps measured no longer measure anything. And half of the “successful” patches would not be merged by a human maintainer.
The dominant variable is no longer the model: at identical model, changing moves scores by 10 to 36 points, the equivalent of a generation. The only evaluation that really concerns you is the one you run on your own code.

A tweet at 11:41 pm

On the evening of July 8, OpenAI’s official account posts a message few vendors would dare write: “We audited SWE-Bench Pro, one of the most used AI coding benchmarks, and found that it no longer reliably measures coding capability at the frontier. We estimate that 30% of its tasks are broken, and we retract our earlier recommendation to use it as a reference evaluation.” The next morning, the same company launched GPT-5.6, its new coding model.

You have to savor the situation in all its depth. SWE-Bench Pro is not an obscure benchmark: it is the one the trade press, comparison tables and discussion threads cite first to separate coding agents. It is also the one OpenAI had officially been recommending since February. And February, precisely: it was already the second time. Five months earlier, the same company had declared the predecessor, SWE-bench Verified, unusable, and advised migrating to Pro. In five months, the sector’s leading lab has therefore declared broken, one after the other, both reference instruments of its own discipline, including the one it had itself crowned.

The temptation is to read a scandal into it, rigged numbers, a maneuver. The reality is more interesting and more uncomfortable: it is a story of metrology. Code benchmarks did not become bad; they went obsolete under the progress of what they measured, and a second variable, long ignored, started to weigh more heavily than the model itself. This report takes the mechanism apart, piece by piece, because it concerns anyone who has ever chosen a model on the strength of a score table.

How a benchmark task breaks

Let us start with what SWE-Bench Pro measures, because the principle is sound. You give the model a real code repository and a real ticket (“this parser mishandles such a case”), it must produce a patch, and hidden tests decide whether the patch is good. No multiple choice, no artificial puzzle: engineering work, judged as in production. On paper, it is the best protocol we know.

The problem is industrial: these tasks are mass-produced, extracted programmatically from real code changes. But a real code change is not designed to become an exam question. OpenAI’s audit sorts the breakages into four families, and each deserves a moment.

The biggest: overly strict tests. The ticket asks to fix a behavior, the hidden tests additionally check implementation details that no one specified. A perfectly correct solution fails because it is not the historical solution. Then comes its mirror image, the under-specified statement: requirements do exist, but only in the tests, invisible from the ticket. The example flagged by the audit is telling: a task from the OpenLibrary project shows a string with a single space in the statement, and the hidden test requires two. No human engineer would pass that kind of test other than by luck. Third family, the exact reverse: tests with too little coverage, which let incomplete patches through. And finally the misleading statement, which actively steers toward the wrong approach.

Note what these four families have in common: none of them talks about the model. They are task-design defects, statements and tests that do not align. This is an important point, because the February diagnosis on Verified was different (we are coming to it): code benchmarks do not all die of the same disease.

The noise ceiling, or why an instrument goes obsolete

OpenAI’s wording deserves a word-by-word reading: SWE-Bench Pro “no longer measures” the frontier reliably. No longer. So it did measure something, for a time. What happened in between? Models improved.

The numbers tell the story better than any speech: on the public set of 731 tasks, the best models went from 23.3% success to 80.3% in eight months. But if roughly 30% of the tasks are broken, the maximum score actually reachable by a perfect agent tops out somewhere around 70%. As long as models struggle at 25 or 40%, this ceiling bothers no one: the gaps between models play out on the valid tasks, and the instrument ranks correctly. But as soon as scores approach the ceiling, everything flips. The points left to win are, for the most part, defective tasks; separating them amounts to measuring which model best guesses the whims of badly designed tests. Like a saturated sensor, the instrument crushes everything above into the same flat zone, and the gaps you read there are noise.

This is the central mechanism of this report, and it is general: a benchmark is not true or false in the absolute, it is usable within a range. The 2026 frontier has simply left the useful range of benchmarks designed in 2024. They were not wrong; they were surpassed, in the literal sense.

The audit itself is worth the detour, for its method and its irony. OpenAI first ran an analysis pipeline over the models’ attempts, their failure traces and the task metadata, with Codex-based investigator agents, able to open the real repositories and run the tests. This pipeline flagged 200 broken tasks (27.4%). In parallel, five experienced engineers ran a human annotation campaign, which identified 249 (34.1%), with 74% agreement between the two approaches, hence the median estimate of about 30%. The method is serious. The irony too: it was Codex agents that audited the benchmark tasked with evaluating Codex. The judged tool now certifies the measuring instrument; you could not better illustrate where the discipline stands.

Three generations of instruments, three deaths: the timeline

Let us pick up the thread from the start, because on one hand the exact scopes of each figure are precisely what the press mangles most, and on the other the July retraction is not the second episode of this story. It is the third.

Instrument	Born	Declared broken	Main cause
SWE-bench (Princeton)	late 2023	2024, by OpenAI	unsolvable or under-specified tasks
SWE-bench Verified (OpenAI)	August 2024, 500 hand-sorted tasks	February 23, 2026	test defects + contamination + saturation
SWE-Bench Pro (Scale AI)	September 2025, anti-contamination design	July 8, 2026	~30% broken tasks (design)

Table 1: Three generations of agentic coding benchmarks, three obsolescences coming ever faster.

At the origin, there is plain SWE-bench, created by Princeton researchers in late 2023: the founding idea, brilliant, of turning real GitHub tickets into exam questions. By 2024, OpenAI finds the benchmark riddled with unsolvable or under-specified tasks, and publishes the remedy: SWE-bench Verified, 500 tasks hand-sorted by professional developers paid to reread each statement. Keep this detail, it gives the rest its true depth: Verified was not one benchmark among others, it was already the corrected version, human-validated, co-built by OpenAI itself. It is that remedy which breaks in February 2026. And SWE-Bench Pro, designed at Scale AI precisely to resist the next ailment ( , with repositories under deterrent licenses and a private set), is the remedy’s remedy. It will have lasted less than a year. Three generations, three ever more careful designs, three ever faster obsolescences: this is not a series of accidents, it is a race between the instruments and what they measure, and the instruments are losing it.

On February 23, 2026, then, OpenAI publishes “Why we no longer evaluate SWE-bench Verified.” The audit does not cover the whole benchmark: it targets the 138 problems (out of 500) that its o3 model repeatedly failed over 64 attempts, the hard subset. Verdict: at least 59.4% of that subset has substantial defects, tests or description, including 35.5% of overly strict tests. Related to the entire benchmark, that comes to at least 16% broken tasks, not 59: the spectacular figure is true, but on a scope that must be cited with it. The symptom that had triggered the audit speaks for itself: model progress there was stalling, from 74.9 to 80.9% in six months, the characteristic flattening of an instrument hitting its limit.

Added to this was an ailment Pro does not have to the same degree: contamination. A paper accepted at NeurIPS, with a limpid title (“The SWE-Bench Illusion”), had shown that the best models recovered the file to fix in 76% of cases from the ticket text alone, without even seeing the repository, against 53% on tasks from repositories outside the benchmark. The models did not always solve the problem; sometimes, they remembered it.

On March 10, the METR evaluation institute adds the missing piece: even when the tests pass, is the work good? Four maintainers of major open-source projects reread 296 AI-generated patches declared “resolved” by the automatic grader. Their merge verdict is about 24 points below the benchmark scores: around 72% of “successful” tasks, about 48% of patches a maintainer would actually accept. Half of the green displayed would not have passed a human code review, for the usual reasons: missed functionality, regression elsewhere, insufficient quality.

And on July 8, then, Pro falls in turn, for task-design reasons this time. Three different diagnoses in five months (contamination, test/mergeable gap, defective tasks), three angles on the same finding: at the frontier, the displayed number and real capability have divorced.

The hidden variable worth a model generation

There is still a floor missing from the explanation, and it is the most counter-intuitive. Grant a benchmark with perfect tasks: would the score obtained finally measure “the model”? Not that either. Because a model never sits a benchmark alone: it sits it in a harness, that gear of tools, system prompt, execution loop and context management we dissected in the report on OpenAI’s comeback. And the harness effect on scores is not a second-order detail. It is a term of the same order of magnitude as the progress of a model generation.

The measurements pile up:

Harness effect: same model, two harnesses, gaps of 14 to 36 points (Success (%))
	Minimal / native harness	Optimized harness
GPT-5.5 (Endor Labs)	61.5	87.2
Opus (independent test)	42	78
deepagents-cli (LangChain)	52.8	66.5

Same model, two harnesses: three independent measurements (Endor Labs on SWE-bench, an independent test on Opus, LangChain on Terminal Bench). The harness moves the score as much as a model generation.

A dedicated research paper (Claw-SWE-Bench) quantifies gaps of up to 27.4 points of success between harnesses, on the same model and the same 350 tasks. Endor Labs measured GPT-5.5 at 61.5% in Codex’s native harness and 87.2% in Cursor’s, the same week: 25.7 points. An independent test swung Opus from 42% with minimal scaffolding to 78% with Claude Code’s full harness: 36 points, without touching the model. LangChain gained 13.7 points on Terminal Bench by changing only its harness, model strictly identical. The trade press settles on a range of 10 to 20 points of routine gap on SWE-bench at constant model, and OpenAI provides an involuntary demonstration in the GPT-5.6 announcement itself: Sol alone gets 88.8% on Terminal-Bench 2.1, and 91.9% when its “ultra” mode coordinates four agents. Same model, plus three points, all it took was changing the orchestration.

And even at a fixed model-harness pair, ordinary variance remains. An agent is not deterministic: from one run to the next, it explores differently, fails or passes borderline tasks. Serious protocols average several random seeds (OpenAI’s cyber evaluations use five per measurement); the press tables, meanwhile, often compare single runs. Two or three points of gap between models, on one run, with no confidence interval: that is not a ranking, it is a draw.

The consequence is brutal for how we talk about these systems: “model X’s score” does not exist. Only the score of a model-harness pair exists, in a given configuration, on a given date, averaged or not. When two tables give incompatible numbers for “the same model”, there is a good chance both are accurate.

That leaves the question of why the instruments go obsolete faster and faster, and the answer lies in an old law of measurement: when a metric becomes a target, it ceases to be a good metric. A public reference benchmark is not a neutral thermometer set beside the market; it is the target, explicit or not, of labs worth hundreds of billions that communicate in points gained. Each training cycle absorbs a little more of the benchmark, through direct contamination of the corpora or through simple selection pressure on what scores. The faster the industry optimizes, the shorter the useful life of a public yardstick. The original SWE-bench did not last a year before requiring its corrected version; Verified lasted eighteen months; Pro, less than a year. The rest is predictable.

What vendors publish, and what to make of it

This landscape casts a particular light on the July 9 communication. Far from skipping benchmarks, the GPT-5.6 announcement lines up the full range: the classic academic ones (GPQA, AIME, MMLU) as well as the agentic ones. It even publishes a figure on that very SWE-Bench Pro the company had just declared broken, and on which Sol actually stays behind Fable 5, 64.6% against 80%. Two anchors, however, call for caution: Artificial Analysis’s index, given as a third party, is an OpenAI pre-evaluation partner, a quality judge more than an independent one; and a footnote specifies that the latencies and costs shown are simulated offline, not measured in production.

The sequence also has to be reported in its exact order, because it feeds a reading you can neither assert nor ignore. On SWE-Bench Pro, Anthropic’s models dominated clearly: 69.2% for Opus 4.8 against 58.6% for GPT-5.5 in the spring. The benchmark where OpenAI was most visibly losing is the one it disavowed on July 8, the eve of its model’s launch. In its favor, the next day’s announcement did not hide the figure: the unfavorable score is there, disavowal included. If there is a maneuver, it is therefore subtler: publish the number and discredit the yardstick in the same gesture. There are the dated facts; the strategic reading remains a hypothesis, and honesty requires saying that the methodological critique, for its part, is solid, corroborated by independent work with nothing to sell (the NeurIPS paper on contamination, the METR note). Add the full irony: the retraction spares no one. Anthropic’s 69.2% rests on the same instrument declared broken; the scores brandished one way or the other now float on the same 30% of defective tasks. No one comes out taller from a warped yardstick.

Measuring yourself

What is left, then? OpenAI’s recommendation fits in one line and it is demanding: benchmarks built directly by experienced engineers, with human supervision integrated from the design stage, not generated in bulk then verified after the fact. Concretely, this looks like the GDPval-type evaluations the company now promotes: tasks written one by one by professionals in the field, with a rubric, judged by humans, at an accepted production cost. It is the admission that quality measurement does not industrialize: what made SWE-bench appealing (thousands of free tasks extracted from GitHub) is exactly what killed it. No successor is anointed for all that; the discipline is starting over from the drawing board.

In the meantime, two families of tools keep their value, precisely because they have internalized the problem. The independent trackers, first: Marginlab runs Codex daily on a contamination-resistant subset of tasks, fifty instances a day, with no custom harness, with a statistical test before declaring a drift; it is this protocol that detected an 8-point degradation of Claude Code in late May, invisible in static benchmarks. Scale SEAL, which we were already citing in the GLM-5.2 report, plays the same role of standardized yardstick. Their value comes not from miraculously perfect tasks, but from the protocol: constant conditions, repeated measurements, significance thresholds. Metrology, in the full sense.

The second family is yours. The teams that come out best have stopped reading other people’s tables and built themselves internal evaluations on their own code: real defects injected into their own pull requests, their languages, their conventions, judged on what a human reviewer would accept. On a smaller scale, the minimal protocol fits in one sentence: before switching model or tool, take five real tasks from your backlog, run them in A/B, measure review time and error rate, and decide on that. Five tasks from your repository are worth more than seven hundred broken tasks from someone else’s.

A method confession, to finish, because it illustrates the reflex to install. Our GLM-5.2 report, published three weeks before the retraction, cited SWE-Bench Pro scores. We had labeled them vendor figures, not replicated, with the independent aggregate index as the only level anchor; that is what lets them age with dignity today: the label survived the instrument. It is exactly the contract that coding-agent measurement now imposes on everyone, vendors, press and readers: each figure with its scope, its date, its harness and its judge. The frontier advances faster than its instruments; there is no reason for that to stop, and the benchmarks that replace SWE-Bench Pro will go obsolete in turn. Measurement is no longer a reading, it has become work again. Yours, preferably.

Sources and method

Editorial freeze date: July 11, 2026. Labels: verified fact (primary source), estimate (third-party analysis), hypothesis (assumed reading).

The July 8 retraction

Fact: OpenAI, Separating signal from noise in coding evaluations (openai.com, July 8, 2026) and official @OpenAI post on X (July 8, 11:41 pm): 731 tasks, pipeline 200 (27.4%), human review 249 (34.1%), 74% agreement, ~30% estimate; four defect families; progress 23.3 → 80.3% in eight months.

The genealogy of the instruments

Fact: SWE-bench: Jimenez et al., Princeton (arXiv 2310.06770, ICLR 2024). SWE-bench Verified: OpenAI, Introducing SWE-bench Verified (August 2024, 500 human-validated tasks). SWE-Bench Pro: Scale AI (September 2025, anti-contamination design).
Fact: OpenAI, Why we no longer evaluate SWE-bench Verified (openai.com, February 23, 2026): 138/500 problems audited, ≥ 59.4% of that subset defective (35.5% overly strict tests), stagnation 74.9 → 80.9%.
Fact: The SWE-Bench Illusion (arXiv 2506.12286, Purdue/Microsoft, NeurIPS 2025): 76% identification of the faulty file from the ticket alone, 53% outside the benchmark, verbatim similarity 35% against 18%.
Fact: METR, Many SWE-bench-passing PRs would not be merged into main (metr.org, March 10, 2026): 296 PRs reread by 4 maintainers, verdict ~24 points below the automatic grader, “golden” baseline 68%.

The harness effect and variance

Fact: Claw-SWE-Bench (arXiv 2606.12344): up to 27.4 points between harnesses at constant model. Endor Labs: GPT-5.5 at 61.5% (native Codex) against 87.2% (Cursor), same week. LangChain, Improving Deep Agents with harness engineering: +13.7 points at fixed model.
Estimate: Opus from 42 to 78% depending on scaffolding (independent test published in April 2026); “10-20 points” range (trade press). The “5-15 points” range going around has no identifiable primary source: we do not repeat it.
Fact: GPT-5.6 announcement (openai.com, July 9): SWE-Bench Pro Sol 64.6% against Fable 5 80%; Terminal-Bench 2.1 at 88.8% (Sol) against 91.9% (ultra mode, 4 agents); latencies and costs “simulated” (footnote 4); cyber evaluations at 5 seeds per measurement; Artificial Analysis pre-evaluation partner.

Scores and trackers

Fact: SWE-Bench Pro: Opus 4.8 at 69.2%, GPT-5.5 at 58.6% (April GPT-5.5 post and trackers); Sol at 64.6% against 80% for Fable 5 (July 9 GPT-5.6 post, published alongside the instrument’s disavowal). Marginlab: daily protocol, N = 50, p < 0.05, 8-point Claude Code degradation detected in late May 2026. Scale SEAL: standardized yardstick (cf. GLM-5.2 report).

Method note. This article reports the July 8-9 sequence (retraction on the eve of launch) as dated facts; any strategic reading is flagged as a hypothesis. The scopes of the percentages (59.4% on Verified’s hard subset; ~30% on Pro’s public set) are cited with each figure, because their confusion is precisely the deformation mode this report documents. This article is the English adaptation of a French original, published on July 13, 2026.

GPT-5.6 and the Codex merger: anatomy of OpenAI's comeback

redaction@lecompute.fr (Christophe Gerardin) — Sat, 11 Jul 2026 00:00:00 GMT

En bref

In forty-eight hours, OpenAI disavowed the benchmark it had been recommending for agentic coding (SWE-Bench Pro, ~30% broken tasks), merged Codex into a single ChatGPT application and launched GPT-5.6 in three pricing tiers. An offensive that turns less on raw intelligence than on price, product and plan.
On the available measurements, GPT-5.6 Sol reaches Claude Fable 5’s level: the lead Anthropic held, which we were still describing in the GLM-5.2 report, has just fallen. That parity is still measured by an OpenAI partner, with no independent replication.
At model parity, the decision turns on economics and the : API price halved, half as many tokens per task, cache retained thirty minutes against five, fast mode inside the plan versus , and an agent orchestration built into the consumer app.

Forty-eight hours to sum up six months

On the evening of July 8, OpenAI publishes a research post that retracts its own recommendation: SWE-Bench Pro, the benchmark the company itself had been advising the community to use since February to evaluate coding agents, is said to contain roughly 30% broken tasks. The next morning, the same company merges Codex into a single ChatGPT application (the old app is renamed “ChatGPT Classic”, the Atlas browser is slated for sunset on August 9), launches GPT-5.6 in three variants, and publishes the full range of scores, down to those of the benchmark it had just declared broken.

Two gestures in forty-eight hours, and the whole story fits inside them. On one side, an admission: at the frontier, public measurement instruments no longer separate much, and we will devote an entire report to this measurement crisis. On the other, a perfectly orchestrated offensive: a model that reaches Anthropic’s level, prices that slip below its own at every tier, and a product overhaul that installs OpenAI’s coding tool at the center of an application claiming 900 million weekly users.

Six months ago, the question barely arose. Among developers, Claude Code dominated the usage and satisfaction surveys, and OpenAI’s models, whatever their published scores, were not convincing hands on the keyboard. Since GPT-5.5 in April, then 5.6 last week, the movement has reversed: teams are migrating, colleagues you thought were settled at Anthropic are switching back to Codex, and the sentiment going around is that you can now switch “without compromise.” This report examines what, in this comeback, belongs to the model, what belongs to price, and what belongs to everything around it. You will see that all three count, but not in the order you might think.

Sol, Terra, Luna: tiers, not sizes

Let us start with the launch itself, because its structure already tells a strategy. GPT-5.6 has been available since July 9, after an open preview on June 26 to about twenty organizations vetted by the US government, an unprecedented arrangement that OpenAI publicly stated its opposition to. The family arrives with a new naming system: the number identifies the generation, and Sol, Terra and Luna designate “durable capability tiers”, each meant to progress at its own pace. Sol is the flagship, Terra the mid-range, Luna the fast and economical tier.

The important word is “tier”, and it is as much about pricing as it is technical. The grid speaks for itself:

Model	Input (€/M tokens)	Output (€/M tokens)	Note
GPT-5.6 Sol	€4.60 ($5)	€27.60 ($30)	= GPT-5.5 pricing
GPT-5.6 Terra	€2.30 ($2.50)	€13.80 ($15)	"competitive with 5.5" at half price
GPT-5.6 Luna	€0.92 ($1)	€5.52 ($6)	the volume tier
Claude Fable 5	€9.20 ($10)	€46.00 ($50)	2× Sol on input, +67% on output
Claude Opus 4.8	€4.60 ($5)	€23.00 ($25)	the former flagship, now affordable

Table 1: API prices per million tokens, July 2026. At each level of need, OpenAI places a model cheaper than the one across the aisle.

That leaves the real question: how good is Sol? On the available measurements, the answer is: the frontier. Artificial Analysis’s Coding Agent index places it at 80.0 against 77.2 for Fable 5, consuming half as many output tokens and for roughly a third less cost per task. On Terminal-Bench 2.1, Sol leads Fable 5 by more than five points. On the same evaluator’s aggregate intelligence index, Fable 5 keeps a one-point lead. One point one way, nearly three the other: at that level of proximity, and given what the instruments are worth today, the right word is parity.

You have to measure what that parity represents. In our GLM-5.2 report, three weeks ago, we were describing Anthropic as an anomaly: Fable 5 then Opus 4.8 towered over the whole pack, with roughly a generation’s lead over the rest of the world. It is that lead which has just fallen, and it is the model event of this launch.

One governance detail is worth the detour: the system card classifies all three tiers, Luna included, at the “High” risk level for cyber and biochem. The economical models in the range are therefore not “lightweight” versions from the standpoint of sensitive capabilities. You pay less, you do not get a throttled model.

What developers are doing, and what to make of it

The adoption figures now, keeping the labels well in sight. OpenAI claims more than 5 million weekly Codex users, “twice as many as three months ago.” It is a vendor figure, with no third-party verification, and the company’s successive snapshots are not perfectly consistent with each other either: 1.6 million in March, more than 4 million on April 21, more than 5 million in early June. The doubling in three months announced in July implies around 2.5 million in mid-April, which sits poorly with the 4 million of the 21st. The order of magnitude and the slope, though, are hardly in doubt; the precision, less so.

Across the aisle, the most solid data predates the wave. The Pragmatic Engineer survey published on March 7, over 906 experienced engineers, gave Claude Code both the most used and the most loved tool, cited by 46% of respondents, twice as much as Cursor and five times as much as Copilot. JetBrains’s January barometer awarded it the best satisfaction in its category. The winter snapshot is unambiguous; it is spring that moved it.

And on perceived quality, be wary of the two corpora in circulation, for symmetrical reasons. The testimonials that ground the classic “split” (Codex cheaper, Claude better on complex code) mostly date from the GPT-5.5 era: the April migrations driven by costs, the comparative tests noting that Codex truncates the middle of long sessions where Claude keeps everything. The impressions on 5.6 are a few days old: Simon Willison finds it “very competent” without being struck by any superiority over Fable on complex tasks, Matt Shumer judges Fable “clearly better” on almost everything he tested. Neither corpus settles Sol’s real quality over the long run. What the author of these lines can add to the file, in the first person and as mere testimony: using both tools daily, the step that still existed last year between what an OpenAI agent and an Anthropic agent produced, at equal instructions, is no longer felt day to day. It is precisely when the difference stops being felt that everything else starts to count double.

The trade-off, piece by piece

Because at perceived parity, economics decide. And that economics does not reduce to the price per million tokens. It plays out in three mechanisms every agent developer knows from their bill, if not in detail.

The first is token efficiency. Sam Altman claims a Sol “54% more token-efficient” on agentic coding tasks; it is an in-house figure, but Artificial Analysis’s measurement points the same way with half as many output tokens as Fable 5 for an equivalent task. On an agentic workload, where the model reasons, calls tools, rereads itself and starts over, the number of tokens emitted weighs more than the unit rate. A model half as talkative at equal price is, from the bill’s point of view, a model half as expensive.

The second is the prompt cache, the mechanism that avoids re-billing the already-sent context on every turn. GPT-5.6 marks an alignment on Anthropic’s model, with writes billed at 1.25× the input rate (a first for OpenAI, it was free before) and reads at 10% of the price. But one parameter changes everything for agents: the minimum cache retention is thirty minutes at OpenAI, against five minutes by default at Anthropic. An agent loop waiting for a test suite to finish or for a human approval finds its cache still there at one vendor, and often pays for it again at the other. It is an infrastructure detail; multiplied by the context re-sent continuously, which makes up the bulk of agent bills, it becomes a budget line.

The third is fast mode, and it is the most revealing, because the technique is identical and the commercial policy opposite. On both sides, “fast” means the same thing: a priority routing in the serving infrastructure, the same model served roughly 1.5 to 2.5 times faster. At OpenAI, this mode burns the subscription’s credits faster (2.5× the standard rate for GPT-5.5), but it draws on the plan. At Anthropic, fast mode, like Fable 5 itself, switches to usage credits billed at the API rate, outside the plan. In one case, accelerating drains the plan faster; in the other, accelerating triggers a second, unpredictable bill. Add the special case of GPT-5.3-Codex-Spark, a distinct model distilled for speed, served on Cerebras’s wafer-scale chips at more than 1,000 tokens per second, reserved for Pro subscribers and rationed according to demand: OpenAI has turned latency into a product, with a range.

It has to be said plainly, because the reverse shortcut is going around: nothing documents that OpenAI’s compute capacity lets it sell speed for less. Fast costs more credits, Spark is rationed. What the plan changes is predictability: a developer knows what they will pay at the end of the month. And if there is one lesson spring 2026 taught teams, it is that the predictability of the bill has become a selection criterion as structuring as model quality.

The spring the subscriptions cracked

That spring deserves to be told in order, because the sequence is eloquent.

Date	Event
April 2	Codex moves from per-message billing to token-aligned billing
May 6	Anthropic doubles Claude Code's 5-hour limits (SpaceX deal, +300 MW)
May 13-14	Anthropic announces programmatic usage will leave the plans on June 15; Microsoft cancels its internal Claude Code licenses; Altman offers 2 months of Codex to enterprises
June 9	Fable 5 launches, included in the subscriptions until June 22
June 12	Fable 5 and Mythos 5 suspended by export-control directive
June 15	Anthropic cancels the plan split, on the very day it was to take effect
June 30 - July 1	Directive lifted; Fable 5 redeployed, included at 50% of the weekly limits
July 8-9	SWE-Bench Pro retraction; Codex/ChatGPT merger; GPT-5.6 launch

Table 2: Three months that redrew the economics of coding agents.

On April 2, OpenAI drops per-message billing for Codex in favor of token-aligned billing: the subscription’s credits now burn in proportion to what the requests actually cost, with a self-reported average cost of €92 to €184 per developer per month ($100 to $200), with high variance. On May 6, Anthropic doubles Claude Code’s five-hour limits and removes peak-hour throttling, leaning on an infrastructure deal with SpaceX for more than 300 megawatts. On May 13, the weekly caps gain 50% until July 13. Generosity climbs on both sides, and it climbs because capacity finally allows it.

Then comes mid-May, and three events in twenty-four hours. On the 13th and 14th, Anthropic announces that programmatic usage of its subscriptions (the agent SDK, the non-interactive command-line interface, the GitHub Actions integrations) will leave the plans on June 15 to shift to a credit pool metered at the API rate. The economic logic is plain: community analyses estimated that some users extracted from their subscription a compute value of twelve to more than a hundred times its price, figures never confirmed by Anthropic but which the head of Claude Code, Boris Cherny, commented on without denying: “our subscriptions were not designed for the usage patterns of these third-party tools.” Also on May 14, The Verge reveals that Microsoft is canceling most of the internal Claude Code licenses in its Windows and Office division, deadline June 30, migration to the in-house tool, under the official banner of “unifying the toolchain.” And the same day, Sam Altman offers two months of Codex free to any company that switches within thirty days. Note the coordination.

The epilogue says everything about the competitive pressure: on June 15, the very day it was to take effect, the pool switch is canceled. Anthropic explains that it is reworking the arrangement to “better support how users build with the subscriptions.” Between the announcement and the cancellation, a month had passed; over the same period, Codex’s in-house counters went from more than 4 million weekly users in late April to more than 5 million in early June.

To grasp what these plans subsidize, the Uber case, documented by Forbes, serves as a benchmark: €138 to €230 per engineer per month on average ($150 to $250), intensive users between €460 and €1,840, adoption up from 32% of engineers in February to 84% in March, roughly 70% of committed code AI-originated, and an annual AI budget exhausted by April. Agentic AI coding is no longer a productivity gadget: it is a first-order budget line, and that is exactly why the subscription war has become the real battlefield.

The Fable 5 aside, or pricing under constraint

One thread of this story deserves its aside, because it shows how much pricing is now decided week by week, under a double constraint of capacity and geopolitics. We told in the GLM-5.2 report the June 12 cutoff: Fable 5 and Mythos 5 switched off worldwide by export-control directive, three days after launch. What that report, frozen on June 21, could not tell is the commercial aftermath.

At the June 9 launch, Fable 5 was included at no extra cost in the subscriptions until the 22nd, with removal announced for the 23rd and a switch to usage credits. The June 12 suspension interrupted the window before its term: the commercial removal never really happened, geopolitics had beaten it to it. Directive lifted on June 30, redeployment on July 1 with a new inclusion window, up to 50% of the weekly limits, until July 7, then extended until the evening of July 12. After which, back to usage credits, and a permanent reintroduction promised “when capacity allows”, with no date. In one month, the best model on the market will have been in turn included, banned, redeployed, re-included at half, extended by five days, then re-billed separately. There is the real state of the market: even Anthropic steers access to its own summit at the mercy of available megawatts.

The harness, or what you are really buying

That leaves the third floor of the comeback, the least visible and perhaps the most decisive: what surrounds the model. A language model never works alone. What turns a text generator into a coding agent is called a harness, and it supplies the model with just about everything that makes its practical effectiveness: the set of tools it can call and their schemas, the proprietary system prompt that sets its behaviors, the project-context injections (the AGENTS.md file at OpenAI, CLAUDE.md at Anthropic), the execution loop with its permissions and its sandbox, the ability to launch subagents (which is, technically, just one tool among others), the hooks that intercept the lifecycle steps, the servers, the skills, and memory management when the context overflows. The model is the engine; the harness is the whole rest of the vehicle. And the result gap between two harnesses, at identical engine, is measured in tens of points on the evaluations: we will devote the next report to it.

On this ground, the two ecosystems have massively converged. Hooks on both sides (with a nuance: Codex’s own up to being a guardrail, their interception is not exhaustive, where Claude Code offers more binding control), subagents on both sides, skills, MCP, context files. Codex even offers an import command that migrates your Claude Code configuration. Differentiation no longer plays out on the feature list, but on the center of gravity: and that is where July 9 changes the game. Native git worktrees, which let you run several independent tasks on the same repository without them stepping on each other, are reserved for the desktop application. The multi-agent command post, session control from the interface, the animated companions that signal at a glance that a session is awaiting approval or has just finished: all of that lives in the app, not in the terminal. The Codex CLI stays maintained, open source, updated the very day before launch; it is simply no longer the storefront. OpenAI bet that agent orchestration is better steered from a graphical interface than from a terminal, and put the result into an application already installed by hundreds of millions of people. Anthropic keeps the favorite tool of extensibility purists; OpenAI has just built the one you do not need to install.

The counter-signals, and what to watch

Honesty requires putting the other pan on the scale, because the first days of 5.6 also carry opposite signals. On Codex’s GitHub repository, users document a perceived quality regression together with rising token consumption since the move to 5.6, others a blow-up in cost per token since mid-June on 5.5. More structurally troubling: Sol’s effective context in Codex is capped around 353,000 tokens, a third of the 1.05 million window sold on the API side. Model parity does not guarantee experience parity, and Marginlab, the independent tracker that measures Codex daily on a contamination-resistant set of tasks, has not yet published a curve for 5.6. The coming weeks will tell whether the regressions are teething troubles or a real step back.

They will also tell something else, because no one believes this snapshot is stable. A single, uncorroborated analyst rumor already puts GPT-6 a few weeks out and makes 5.6 the last of its line; Anthropic is preparing the next Fable. We have dated this report to July 11 and will keep the update box, because in this cycle, eight weeks are enough to overturn a conclusion.

What will probably survive the next launch, on the other hand, is the shift this spring brought about. The question “which is the best model” has just lost its monopoly. A developer choosing their tool in July 2026 is arbitrating a complete system: a model, a harness, a plan policy, a price curve, and the trust they place in the figures that claim to compare them. OpenAI did not win this round by becoming the smartest again. It won it by making the comparison undecidable on the model, then making it very decidable on everything else. The next round will play out by the same rules, and nothing says there will not be, this time again, a reversal.

Sources and method

Currencies converted at the indicative rate of 1 USD = 0.92 EUR (mid-2026). Editorial freeze date: July 11, 2026; an update box will be added if the facts change. Labels: verified fact (primary source), estimate (third-party analysis), self-reported (vendor figure with no independent verification), hypothesis (rumor or assumed reasoning).

GPT-5.6 launch & July 9 merger

Fact: Announcement post (openai.com/index/gpt-5-6, July 9, 2026): Sol/Terra/Luna tiers, API prices, cited evals; Sol preview (openai.com/index/previewing-gpt-5-6-sol, June 26); system card (all three tiers “High” cyber/bio).
Fact: Desktop merger and “ChatGPT Classic”: OpenAI help center, Moving to the new ChatGPT desktop app (July 9); Atlas sunset announced July 9 for August 9 (James Sun, OpenAI, on X).
Self-reported: Codex adoption (1.6M in March; 4M+ on April 21; 5M+ on June 2; “×2 in three months”, r/Codex AMA of July 10): OpenAI figures, inconsistencies noted in the body of the article.

Parity & benchmarks

Estimate: Artificial Analysis (Coding Agent Index v1.1, Intelligence Index, readings of July 9-10): OpenAI pre-evaluation partner, flagged as such; latencies and costs in the OpenAI announcement “simulated” (footnote 4 of the post).
Fact: Retraction of the SWE-Bench Pro recommendation: openai.com/index/separating-signal-from-noise-coding-evaluations (July 8) and @OpenAI post on X: ~30% broken tasks out of 731, pipeline 27.4%, human review 34.1%. The next day’s 5.6 post nonetheless publishes the full range of scores, including SWE-Bench Pro: Sol 64.6% against 80% for Fable 5 (re-reading of the post, July 11).
Fact: The Pragmatic Engineer survey (March 7, 2026, n = 906); JetBrains AI Pulse (January 2026). Dated testimonials: Simon Willison (July 9), Matt Shumer (X, via Axios, July 9).

Pricing, plans & cache

Fact: Codex fast mode and billing: developers.openai.com/codex/speed and /codex/pricing (fast ×1.5, credits ×2.5 on GPT-5.5, ×2 on 5.4; token switch of April 2, rate card); GPT-5.6 prompt caching (writes 1.25×, minimum retention 30 min).
Fact: On the Anthropic side: claude.com/pricing and support.claude.com (fast mode and Fable 5 in usage credits; cancellation of the pool split on June 15, article 15036540); 5-hour limits doubled and SpaceX deal (Anthropic, Higher usage limits for Claude and a compute deal with SpaceX, May 6); weekly +50% (May 13).
Estimate: “twelve to more than a hundred times”: community analyses (including the OpenClaw case documented by The Register), never confirmed by Anthropic; Boris Cherny quotes (The Register, VentureBeat).
Fact: Microsoft: Tom Warren scoop (The Verge, Notepad newsletter, May 14); OpenAI response the same day (2 months of Codex free, Altman on X). Uber case: Forbes, Uber Burns Its 2026 AI Budget In Four Months On Claude Code (May 17).

Fable 5 saga

Fact: Anthropic: launch and inclusion window (anthropic.com/news/claude-fable-5-mythos-5, June 9); June 12 suspension (see our GLM-5.2 report for the full timeline); Redeploying Claude Fable 5 (June 30); extension to July 12 (official channels, July 7).

Harness & counter-signals

Fact: Codex docs: worktrees (developers.openai.com/codex/app/worktrees), hooks, companions; github.com/openai/codex (Apache-2.0; issues #32161 5.6 regression, #31860 effective context ~353K, #28879 cost per token). Claude Code docs: code.claude.com/docs (hooks, subagents, skills, plugins).
Fact: Marginlab: independent daily tracker, no GPT-5.6 curve as of July 11.
Hypothesis: GPT-6 “a few weeks out”, 5.6 “last of the series”: single analyst tweet (@synthwavedd, ~July 7), uncorroborated.

Method note. This article was written forty-eight hours after the launch of GPT-5.6. OpenAI’s adoption and efficiency figures are given as self-reported as long as no third-party measurement has replicated them; the level anchor used is Artificial Analysis’s index, whose commercial tie with OpenAI is flagged. Quality testimonials are systematically dated, the bulk of the public comparative corpus dating from the GPT-5.5 era. This article is the English adaptation of a French original, published on July 11, 2026.

DGX Spark vs RTX 5090: capacity or speed, the choice that decides your local LLM

redaction@lecompute.fr (Killian Pluenet) — Wed, 08 Jul 2026 00:00:00 GMT

Two objects with nothing in common, one buying question

On one side, a 575 W card that demands a case, an oversized power supply and serious airflow. On the other, a golden cube the size of a book, complete, silent, sitting next to the monitor. Everything separates them except the moment you price them: with the shortage pushing the RTX 5090 far above MSRP (from ~$3,000 on the US street, €3,900 to €4,400 in Europe), the DGX Spark at $4,699 now plays in a comparable envelope. At overlapping budgets, you have to choose. And the right criterion is neither the petaflop nor the core count: it is the shape of your workload.

The whole duel reduces to one trade-off: memory eliminates, bandwidth ranks. Here are the two spec sheets side by side, then what the measurements say.

	DGX Spark (GB10)	RTX 5090 (desktop)
Memory	128 GB unified LPDDR5X	32 GB GDDR7
Bandwidth	~273 GB/s	~1,792 GB/s (6.5×)
CUDA cores	6,144 (RTX 5070 class)	21,760
Power	whole machine, 240 W max (~170 W measured under load)	card alone, 575 W
Form factor	standalone mini PC (Linux DGX OS)	a card to integrate (full PC required)
Observed price	$4,699 (official MSRP)	$2,999+ US street, €3,900–4,400 EU ($1,999 MSRP)

Table 1: The duel in numbers. Vendor specs (verified facts); the DGX Spark bandwidth comes from the GB10 datasheet, 5090 street prices observed July 2026.

What the measurements say: bandwidth decides

The mechanism first, because it explains every line of measurement. LLM generation re-reads the entirety of the model’s weights for every produced token: output throughput is capped by how fast memory can be traversed, almost never by compute. The 5090 traverses its 32 GB at ~1,792 GB/s; the Spark traverses its 128 GB at ~273 GB/s. On the same model, the throughput ratio follows the bandwidth ratio, and that is exactly what the LMSYS measurements on the DGX Spark show: on GPT-OSS 20B, ~205 tokens/s of generation for the 5090 against ~50 for the Spark. Four times less, for a bandwidth ratio of 6.5: the measured gap is a little softer than raw physics, because with so few active weights per token the 5090 no longer saturates its bandwidth (fixed runtime costs take over), while the Spark stays pinned to its memory ceiling.

(prompt ingestion) is the exception that proves the rule: that phase is compute-bound, and the Spark’s FP4 peak holds its own there. But an interactive session lives in , and decode reads the memory line of the spec sheet, not the line.

The verdict by model size

Up to ~30B: the 5090, no contest. A 30B in (~17 GB) fits in 32 GB with room for the , and GDDR7 serves it at dozens of tokens/s. The Spark runs the same models four times slower, and nothing offsets that gap in interactive use. Same conclusion for 20B models: measured by LMSYS, the gap stays one to four.

Dense 70B: the gap between the two. In Q4_K_M, its ~42 GB of weights overflow the 5090, which must offload layers to CPU RAM at collapsed throughput. The Spark hosts it effortlessly, then serves it at the pace its memory allows: a ~4 tokens/s physical ceiling in dense FP8, 2.7 measured by LMSYS once runtime overhead is paid. Neither machine is right for fast dense 70B; that need starts at 48 GB of fast memory, the territory of the RTX PRO 6000, dual GPUs, or rented datacenter cards.

100B+ MoE: the Spark, by forfeit. A GPT-OSS 120B (~65 GB of MXFP4 weights, ~80 GB loaded) does not fit in any consumer card, 5090 included. The Spark loads it and serves it at ~38 tokens/s measured in llama.cpp (~50 with SGLang: the figure is runtime-dependent), because a MoE only reads a fraction of its weights per token: capacity pays, bandwidth strangles less. It is the only segment where the duel has a single contender, and it is precisely what the Spark exists for.

The extended duel: RTX 5070, RTX 5090 Laptop and RTX Spark

Searches attach other names to this duel, and dismissing them properly avoids several buying mistakes. The first hides in the name itself: the RTX 5090 Laptop is not a shrunken desktop 5090, it is a different chip (GB203, the desktop 5080’s die) with 24 GB of GDDR7 at ~896 GB/s and 10,496 cores within 95 to 150 W. For LLMs, read it as a very good 24 GB card, half the speed of the desktop 5090; every “5090” figure in this article refers to the desktop card. The RTX 5070 shares its CUDA core count (6,144) with the Spark’s GPU, which makes it the “compute equivalent” on paper; but its 12 GB of GDDR7 (~672 GB/s) confine it to ~13B models in 4-bit. It is an honorable entry into local LLMs, not a Spark rival: you do not compare 12 GB to 128. The RTX Spark (N1X), expected in fall 2026 on Windows on Arm, is the DGX Spark’s consumer cousin: same 128 GB unified pool, same ~~273 GB/s wall. Against the 5090 it will replay this article’s duel exactly, with a probably lower entry price; the analyst estimates in circulation (~~$2,899) cover its 16-32 GB entry configurations only.

How to decide

Ask the model question before the machine question. Does your largest model fit in 32 GB, weights and KV cache included? If yes, the 5090 is objectively superior: comparable money, four times the throughput. Are you targeting a 100B+ MoE locally? The Spark is the only contender under $5,000, and the experience stays comfortable as long as the model is MoE. Are you in between, on fast dense 70B? Neither: move up to fast 48 GB+ memory, or rent by the hour until you have measured your real need.

Conclusion

This duel has no winner; it has two winners on disjoint terrain, and a no man’s land in between. The 5090 is the machine for mid-size models served fast; the Spark is the machine for the big MoE you simply could not run at home before it. What to watch next: fall 2026, when the RTX Spark replays this match at a lower price, and the LPDDR6 generation, the only thing capable of moving the 273 GB/s wall that defines the capacity camp today. The day a unified SoC crosses 500 GB/s, this duel changes nature; until then, it is decided by model size.

Sources and method

This article is the “Spark versus GeForce” spoke of our Spark coverage: the shared figures (GB10 datasheet, measurements, prices) are established and sourced in RTX Spark vs DGX Spark and were re-verified on July 4, 2026. A French version of this article, published July 8, 2026, is the original.

Verified facts. DGX Spark bandwidth and memory (~273 GB/s, 128 GB LPDDR5X, 256-bit bus): DGX Spark / GB10 datasheet. RTX 5090 (32 GB GDDR7, ~1,792 GB/s, 21,760 cores, 575 W) and RTX 5070 (12 GB, ~672 GB/s, 6,144 cores): NVIDIA spec sheets. DGX Spark MSRP at $4,699: raised from $3,999 in February 2026 (Tom’s Hardware, VideoCardz). GPT-OSS 120B weights (~65 GB MXFP4, ~80 GB loaded): OpenAI model card. DGX Spark power: 240 W peak system, GB10 SoC at 140 W TDP (NVIDIA hardware guide); the ~170 W under AI load is a press observation (StorageReview, NVIDIA forums), workload-dependent.

Measurements (proxy). GPT-OSS 20B throughput (5090 ≈ 205 tokens/s; DGX Spark ≈ 50), GPT-OSS 120B on Spark (≈ 38 tokens/s in llama.cpp, ≈ 50 in SGLang) and Llama 3.1 70B FP8 on Spark (2.7 tokens/s decode): LMSYS review of October 13, 2025 and community benchmarks, detailed in the Spark deep dive. Measured under Linux (DGX OS); read as orders of magnitude.

Credible estimates. RTX 5090 street prices: ~~$2,999+ on the US secondary market (gpudrip, videocardz) and €3,900-4,400 in EU retail (bestvaluegpu, VideoCardz), observed early July 2026 in a DRAM-shortage market. RTX Spark N1X (~~$2,899 for 16-32 GB entry configurations): Morgan Stanley estimates, June 2026, unconfirmed by NVIDIA.

Stated assumptions. The ~4 tokens/s ceiling for dense 70B FP8 on the Spark is the division 70 GB ÷ 273 GB/s (memory ceiling, single sequence); the matching LMSYS measurement is 2.7 tokens/s, below the ceiling as expected (runtime overhead, KV cache traffic). The ~4× throughput ratio between the 5090 and the Spark is measured on GPT-OSS 20B; it varies with model, runtime and batch. The LPDDR6 projection and the ~500 GB/s threshold in the conclusion are trend hypotheses, not a confirmed roadmap.

Image credit. Header photo: KXRORS S300 SFF PC with GeForce RTX 5070 Founders Edition next to NVIDIA DGX Spark by Daniel Lu, CC BY-SA 4.0, via Wikimedia Commons, cropped.

Prefill/decode disaggregation reaches production: state of play, May 2026

redaction@lecompute.fr (Killian Pluenet) — Fri, 03 Jul 2026 00:00:00 GMT

Key takeaways

(splitting the inference pipeline into specialized prefill and decode node pools) moved from paper to production in three months: Dynamo 1.0, SGLang 0.5.12, and TensorRT-LLM 1.3 each implement disaggregated serving with native KV transfer.
Tenstorrent poses the hardware counter-argument: Galaxy Blackhole unifies compute and memory in a system with 6.2 GB of SRAM and 2.9 PB/s of internal bandwidth, reaching 350+ tokens/s/user in decode on DeepSeek-R1 671B without disaggregation.
MLPerf v6.0 validates the software lever: the cycle’s most spectacular gains come from orchestration, not silicon. NVIDIA improves its DeepSeek-R1 score by 2.7× over its own earlier submission, on identical hardware.

Where things stood

On May 17, 2026, our article on the Vera Rubin platform laid out a thesis: LLM inference is no longer a homogeneous GPU problem. , the phase that processes the whole prompt at once, is compute-bound: it saturates the . , the token-by-token generation, is memory-bound: it waits on , the GPU’s high-bandwidth memory. Bolting the two onto the same GPU underuses the hardware on one of the two phases. NVIDIA acknowledged this at the silicon level by integrating a dedicated Groq LPU for decode into the Vera Rubin platform, and by launching Dynamo 1.0 as the orchestrator.

A week later, the thesis is no longer an architectural projection. Three software stacks offer disaggregation in production. A hardware challenger disputes the principle itself. And the first MLPerf cycle of the disaggregation era measures what software is worth against silicon. This is the state of play this article surveys.

The disaggregated pipeline in one diagram

Request User prompt Input

Prefill Tensor Cores saturated, compute-bound Dedicated GPU nodes

KV transfer RDMA / NVLink / NIXL / MORI-IO Critical point

Decode HBM bandwidth, memory-bound GPU or LPU nodes

Response Generated tokens Output

Simplified disaggregated pipeline. The KV transfer between prefill and decode is the critical point: this is where NIXL, Mooncake, and MORI-IO come in.

The difficulty of this pipeline lives in the third column. The , the attention keys and values the model computes from the prompt and reuses for every later token, is produced by the prefill node and must reach the decode node before generation can start, and that transfer is measured in gigabytes per request. On a 70B model in , an 8k-token context amounts to ~1.3 GB of KV cache. On a DeepSeek-R1 671B with a 100k context, it runs to tens of gigabytes. If the transfer is slower than decode itself, disaggregation degrades latency instead of improving it. Each runtime solves this its own way.

Dynamo 1.0: the orchestrator in production

NVIDIA shipped Dynamo 1.0 as a production release on March 16, 2026, the same day as the Vera Rubin announcement. That timing is deliberate: Dynamo is the software link between the hardware thesis (silicon specialized per phase) and its execution.

What Dynamo does. It is an inference orchestration framework, open source (GitHub: ai-dynamo/dynamo), that coordinates a cluster of prefill and decode nodes. It routes requests in real time, transfers the KV cache between nodes, and manages the elasticity of the pool. Its native mode is disaggregated serving: the prefill nodes compute the prompt, the KV cache is transferred over (NVIDIA’s chip-to-chip interconnect) or RDMA to the decode nodes, and generation starts without the prefill node being blocked.

What sets it apart from the runtimes. Dynamo is not an inference runtime: it relies on TensorRT-LLM or SGLang as the execution backend. Its role sits above: KV-aware routing, topological placement across the cluster, and above all DGDR (Dynamo Graph Deployment Request), a mechanism that translates SLOs (target latency, minimum throughput) into a concrete deployment on the cluster. Where a runtime answers “how do I serve this model on this GPU”, Dynamo answers “how do I spread this model across 72 GPUs to hold these constraints”.

Multimodal E/P/D. Since March 2026, Dynamo extends the model to three phases (Embedding, Prefill, Decode) with a dedicated embedding cache. On image requests with Qwen3-VL-30B-A3B-Instruct in FP8 on GB200, NVIDIA measures a 30% reduction in TTFT (time to first token) and a 25% throughput gain (verified fact, NVIDIA source). The extension to video is underway: FastVideo plus SGLang Diffusion generates 5 seconds of video in ~40 seconds on a single Hopper GPU.

Kubernetes integration. The K8s Inference Gateway plugin implements KV-aware routing directly inside Kubernetes infrastructure: requests that share a KV prefix are routed to the node that already holds it. NVIDIA measures a 20× faster TTFT and a 4× faster end-to-end latency on requests with a common prefix (verified fact). Grove (GitHub: ai-dynamo/grove) rounds out the picture with topology-aware scheduling for GB300 NVL72: it places workloads according to the rack’s NVLink topology.

Adoption. The list of confirmed adopters as of May 24, 2026 includes AstraZeneca, Baseten, ByteDance, CoreWeave, Crusoe, DigitalOcean, Gcore, GMI Cloud, Nebius, Meituan, Pinterest, Prime Intellect, Rednote, SoftBank Corp., Tencent Cloud, Together AI, and Vultr. Cloud integrations: AWS, Azure, GCP, OCI, Alibaba Cloud.

The Baseten case: the nuanced numbers

Baseten published a field report as early as October 2025, as an early adopter. The post’s title (“How Baseten Achieved 2x Faster Inference with NVIDIA Dynamo”) sums up a TTFT cut in half. The detailed measurements, by Abu Qader, Michael Feil, and Rachel Rapp, are more granular: a 50% reduction in TTFT, a 61% increase in requests per second, and a 62% increase in tokens per second on Qwen3 Coder 480B with prompts of about 50k tokens. The “2×” applies to TTFT, not to overall throughput. And the “no additional hardware cost” claim comes from social media, not from the technical post.

The aggregate throughput figure NVIDIA cites (7× throughput with disaggregation plus extended Expert Parallelism on GB200 NVL72, measured on DeepSeek R1-0528 in , 1k/1k context) comes from a SemiAnalysis InferenceX benchmark dated March 3, 2026, relayed by NVIDIA. It is a vendor-cited figure: representative of an optimized case, not of an average production throughput.

SGLang 0.5.12: native disaggregation

SGLang 0.5.12 shipped on May 16, 2026, and it is the first version of the runtime to treat prefill/decode disaggregation as a first-class citizen. Where Dynamo is an orchestrator above the runtime, SGLang builds the machinery into the runtime itself.

The mechanism. The GPU Staging Buffer is an intermediate VRAM buffer that accumulates the KV cache on the prefill side and transfers it block by block to the decode node. The Decode Radix Cache on the decode side organizes the received KV in a radix tree that enables prefix matching: if two requests share the same first 2,000 tokens, the cache is transferred only once. Three transport backends are supported: NIXL (NVIDIA), Mooncake (an open-source framework), and MORI-IO (AMD), which makes SGLang the only runtime to natively support disaggregation on both GPU ecosystems.

DeepSeek V4 day-0. SGLang 0.5.12 is the first runtime to support the full DeepSeek V4 inference path from launch day. This matters: DeepSeek models use a MoE (Mixture of Experts) architecture with a very large number of experts, and serving them efficiently demands Tensor Parallelism, Expert Parallelism, Context Parallelism, and Data Parallel Attention at once. SGLang supports TP16 on H100 and H20.

HiSparse: proactive offload of idle KV. HiSparse is a mechanism specific to sparse-attention models of the DeepSeek family (V3.2, GLM-5.1). Instead of waiting for VRAM to saturate before starting the offload, HiSparse proactively identifies the KV entries that no longer participate in attention and moves them to CPU memory. The measured gain: 3× throughput at 256 concurrent requests, up to 5× on long contexts. The limit: it works only on sparse-attention models (DSA). On a classic dense-attention Llama, the mechanism does not apply.

HiCache complements HiSparse by adding a unified radix tree (UnifiedRadixTree) that tiers the KV cache across VRAM, CPU RAM, and SSD via Mooncake. It extends the logic of PagedAttention, which pages the KV cache the way an operating system pages virtual memory, to three storage levels instead of two.

Native Blackwell. The TokenSpeed MLA backend is optimized for Blackwell Tensor Cores with an FP8 KV cache. The W4A4 MegaMoE kernels target 4-bit-quantized MoE models, and V2, where a lightweight draft path proposes several tokens that the full model then verifies in one batch, improves the acceptance rate on MTP (Multi-Token Prediction) models. DeepSeek-R1 ships a native MTP layer, extended to 3 for the performance peak in Dynamo plus TRT-LLM.

TensorRT-LLM 1.3: Helix Parallelism and the KV connectors

TensorRT-LLM 1.3 has been in a fast pre-release cycle since March 2026: rc7 on March 10, rc12 on April 17, rc15 on May 21. Each release candidate adds a piece of the disaggregation puzzle.

Service discovery for disaggregated serving (rc12): prefill and decode nodes discover each other automatically on the cluster, with no static configuration. It is the prerequisite for Dynamo to route requests without a hard-coded registry.

KV Cache Connector API (introduced in v1.1, refined in the 1.3 release candidates): an abstraction API for KV transfer between nodes. The runtime does not need to know how the KV is transferred (NVLink, RDMA, S3/Azure blob): the connector handles it. Dynamo uses this API to offload KV to object storage, a feature shipped in March 2026.

Sparse Attention (rc13): support for MQA and GQA sparse attention, needed for DeepSeek models and sparse-attention architectures.

Helix Parallelism: the central mechanism

Helix Parallelism is TRT-LLM 1.3’s most significant contribution to disaggregated serving. Its mechanism deserves the detail, because it solves a problem that existing parallelism strategies handled poorly.

Here is the problem. On a MoE model like DeepSeek-R1 671B, each transformer layer consists of two blocks: attention (which is dense, every parameter participates) and the FFN (feed-forward network, which is sparse, only a few experts fire per token). Classic Tensor Parallelism splits all operations uniformly across the GPUs. On dense attention, that is optimal: each GPU does its share. On the sparse FFN, it is waste: the GPUs hosting inactive experts sit idle, while those hosting the hot experts saturate.

Helix resolves this imbalance by applying two different parallelism strategies depending on the block. During attention, it uses KV Parallelism: each GPU holds a partition of the KV cache and computes its share of attention independently. During the FFN, it switches to TP (Tensor Parallelism) for dense models, or TP×EP (Tensor Parallelism × Expert Parallelism) for MoE models. This dynamic switching between two parallelism regimes within a single layer (hence the name Helix, which evokes interleaving) is what sets the approach apart. The paper (arXiv: 2507.07120) measures up to a 1.5× reduction in TTL (Token-to-Token Latency) and the ability to support 32× larger batches under the same latency constraint, on DeepSeek-R1 on Blackwell.

It is a software answer to the same problem that hardware disaggregation attacks on the silicon side: match the execution regime to the workload profile, layer by layer.

RocketKV: compression without training

RocketKV (arXiv: 2502.14051, ICML 2025, NVlabs) is a two-stage KV compression technique integrated into TRT-LLM 1.3. The first stage, SnapKV++, evicts the KV entries that contribute little to attention, by measuring their accumulated importance score. The second stage applies a hybrid top-k sparse attention on the remaining entries. The gain measured on H100: a 3× decode speedup, a 31% reduction in peak memory use. The distinctive point: no retraining is required. The compression applies at inference, on an existing model.

The counter-argument: Tenstorrent Galaxy Blackhole

All the solutions above start from one premise: disaggregation is necessary because the general-purpose GPU is structurally sub-optimal on one of the two phases. Tenstorrent disputes that premise.

On April 28, 2026, Tenstorrent launched Galaxy Blackhole into general availability. The positioning is explicit in their press release: “Other solutions require bolting together separate accelerators across fragmented infrastructure.” Their thesis: rather than fragmenting the pipeline across different silicon linked by a network, you unify compute, memory, and network in a single system built so that the internal bandwidth is high enough that the prefill/decode distinction loses its relevance.

The specs behind the thesis. Galaxy Blackhole bundles 32 Blackhole chips into a single system. Compute reaches 23 PFLOPS in Block FP8. Memory is split across two tiers: 6.2 GB of SRAM at 2.9 PB/s of internal bandwidth, so that the SRAM alone offers a bandwidth on a different scale from the HBM of a classic GPU. For reference, an H100 SXM5 tops out at ~3.35 TB/s of HBM3, a B200 at ~8 TB/s of HBM3E. Above that sits 1 TB of DRAM at 16 TB/s. The internal network supports up to 56 ports of 800G Ethernet.

Tenstorrent’s reasoning runs as follows. The reason decode is memory-bound on a classic GPU is that HBM bandwidth (~8 TB/s on a B200) cannot keep pace with the compute. If you replace the GPU-plus-HBM hierarchy with an SRAM-first hierarchy at 2.9 PB/s, the arithmetic intensity of decode rises enough that compute becomes the limiting factor again, not memory. The result: the prefill/decode split loses its purpose. A single system does both efficiently.

Blitz Mode makes this thesis concrete. On DeepSeek-R1-0528 (671B parameters), Tenstorrent claims: 350+ tokens/s/user in decode, TTFT under 4 seconds on a 100k-token context, batch sizes from 8 to 64, context up to 128k. These are vendor figures (a verified fact as to their publication, not verified by an independent benchmark), but the sub-4-second TTFT on 100k tokens is an aggressive claim: it means the prefill itself is fast despite the absence of dedicated Tensor Cores in NVIDIA’s sense of the term.

The price. A Galaxy Blackhole system starts at around $110,000. A 4-system supercluster starts at around $440,000. Set against the cost of a DGX GB200 NVL72 rack (press estimates put it between $2 million and $3 million), that is an order of magnitude below. The direct comparison stays misleading, because the two systems do not target the same model scale or the same inter-rack network bandwidth.

AMD: MORI-IO and vLLM-ATOM

AMD has no orchestration framework equivalent to Dynamo. Its strategy is to supply the transport building blocks that third-party runtimes consume, and to make sure its GPUs are first-class citizens in SGLang and vLLM.

MORI-IO (Modular RDMA Interface) is AMD’s point-to-point communication library for low-overhead KV transfers (GitHub: ROCm/mori). It is not an inference runtime: it is an RDMA transport layer optimized for disaggregation use cases, namely the transfer of KV cache between prefill and decode nodes (the Expert Parallelism communication of MoE models falls to a separate module, MORI-EP). MORI-IO is integrated into SGLang (as a transport backend alternative to NIXL) and into vLLM. It is the piece that makes disaggregation possible on AMD GPUs, exactly as NIXL makes it possible on NVIDIA.

vLLM-ATOM is an external plugin published on May 7, 2026, that bundles AMD’s kernel optimizations for vLLM: AITER fused attention, custom AllReduce, optimized MoE routing, FP4 support for MI355X. AMD describes it as a “temporary proving ground for new optimizations”; the stabilized kernels are upstreamed into vLLM’s native ROCm backend. The logic is pragmatic: rather than wait on upstream, AMD ships the optimizations in a plugin you install alongside, one that will disappear once upstream has absorbed them.

MI350P PCIe deserves a mention. Announced on May 7, 2026, it is the first AMD MI350 accelerator in PCIe Gen 5 form factor: 144 GB of HBM3E at 4 TB/s, up to 4.6 PFLOPS in MXFP4, passive dual-slot, 600 W TBP (configurable down to 450 W). It is not the same class as the MI355X (288 GB, 8 TB/s): it is a drop-in for existing air-cooled servers, designed for companies that want to enter disaggregated inference without overhauling their cooling infrastructure.

What MLPerf v6.0 measures

MLPerf Inference v6.0, published on April 1, 2026 by MLCommons, is the first benchmark cycle that reflects the era of software disaggregation. 24 organizations submitted results. Five new benchmarks were added: GPT-OSS 120B, interactive DeepSeek-R1 (with speculative decoding), DLRMv3, WAN-2.2 text-to-video, and YOLOv11 edge.

The most striking result is not a hardware number: it is a software number. NVIDIA improved its DeepSeek-R1 score by 2.7× over its own earlier submission. The hardware did not change: 288 GPUs, 4 GB300 NVL72 racks. The gain comes from orchestration: Dynamo, disaggregation, Expert Parallelism, extended speculative decoding (MTP raised from 1 to 3 layers). The result: 2.5M tokens/s. This 2.7× has to be read precisely: it is against the earlier NVIDIA submission, not against the whole MLPerf v5.1 field. It shows that software weighs as much as silicon on already-deployed hardware.

AMD MI355X. The flagship score is 100,282 tokens/s on Llama 2 70B Server, 3.1× the MI325X. In multinode (11 nodes, 87 MI355X), AMD reaches 1,042,110 tokens/s Offline and 1,016,380 tokens/s Server. These figures confirm the MI355X’s competitiveness in raw throughput, even if AMD’s software ecosystem stays a notch behind on orchestration maturity.

NVIDIA Blackwell Ultra reaches 2.5M tokens/s on DeepSeek-R1 with 288 GPUs spread across 4 GB300 NVL72 racks, the highest absolute score of the cycle. That this score comes largely from software gains on existing hardware is the most important signal of this MLPerf cycle: investing in orchestration pays as much as investing in silicon.

The implementations compared

Criterion	Dynamo 1.0	SGLang 0.5.12	TRT-LLM 1.3	Tenstorrent Galaxy
Role	Cluster orchestrator	Native runtime	Runtime + kernels	Integrated system
Disaggregation	Native (E/P/D)	Native (P/D)	Via Service Discovery + KV Connector	Not applicable (unified)
KV transport	NVLink, RDMA, S3/Azure blob	NIXL, Mooncake, MORI-IO	KV Cache Connector API	Internal SRAM (2.9 PB/s)
Supported GPUs	NVIDIA (GB200, GB300)	NVIDIA + AMD (H100, H20, MI355X)	NVIDIA only	Blackhole (RISC-V Tensix)
Advanced parallelism	Wide EP, TP, KV-aware routing	TP, EP, CP, DPA	Helix (KV-P + TP×EP)	Internal, 32 chips
KV optimization	KVBM offload S3, prefix caching	HiSparse, HiCache, Radix Cache	RocketKV, Sparse Attention	SRAM-first (6.2 GB)
Kubernetes	Inference Gateway + Grove	No	No	No
Open source	Yes (ai-dynamo/dynamo)	Yes (sgl-project/sglang)	Yes (NVIDIA/TensorRT-LLM)	No
Entry price	Cloud GPU (~$2-3/h H100)	Cloud GPU (~$2-3/h H100)	NVIDIA GPU only	~$110,000 (system)

Table 1: prefill/decode disaggregation implementations, May 2026. The columns compare the approach, KV transport, supported models, and positioning.

How to choose

The decision matrix depends on three variables: your existing hardware fleet, your dominant constraint (latency or throughput), and your tolerance for operational complexity.

You have an existing NVIDIA fleet and you serve MoE models at high concurrency. Dynamo plus TRT-LLM is the most direct path. Dynamo orchestrates the disaggregation, TRT-LLM delivers the maximum per-node throughput with Helix Parallelism, and the Kubernetes plugin handles scaling. The operational cost is real (one more layer to maintain), but the measured gains (7× throughput on the NVIDIA benchmark, 2× TTFT at Baseten) justify the investment on high-concurrency workloads.

You want disaggregation without hardware lock-in. SGLang 0.5.12 is the only runtime that supports disaggregation natively on NVIDIA and AMD, via NIXL and MORI-IO respectively. If you plan a mixed fleet or a future migration to AMD, it is the most defensible choice. HiSparse is a powerful lever if you serve DeepSeek models.

You start from scratch and per-user latency is your key metric. Tenstorrent Galaxy Blackhole is worth the evaluation. The sub-4-second TTFT on 100k tokens and the 350+ tokens/s/user in decode are aggressive claims, but the entry price (~$110,000 per system) and the absence of KV-transfer complexity are real arguments. The trade-off: a young software ecosystem, no native Kubernetes, and a lock-in to the Tensix architecture.

You serve mostly dense models (Llama, Mistral) at moderate batch. Disaggregation is probably not your immediate priority. A well-configured classic runtime, vLLM with and prefix caching, covers the majority of these cases. Disaggregation pays off mainly on MoE models at very high concurrency or on long contexts.

Conclusion

Software caught up with hardware in three months. The thesis Vera Rubin laid out, that inference breaks into specialized phases, is now implementable without waiting for Rubin silicon. Dynamo, SGLang, and TRT-LLM let you disaggregate on existing Blackwell or Hopper hardware, and the measured gains (MLPerf v6.0: 2.7× from software alone) show that orchestration has become a lever at least as powerful as the jump to the next silicon generation.

The next frontier is not disaggregation itself: it is its automation. The three runtimes still demand manual sizing of the prefill/decode ratio, explicit configuration of the KV transport, and a network topology designed for the use case. Dynamo’s DGDR sketches the direction: translate an SLO into a deployment, without the operator touching the routing. Once that loop closes, an SLO goes in and a configured cluster comes out, disaggregation will stop being a specialist’s technique and become a default deployment mode.

The other open question is the fourth tier. Vera Rubin laid out three tiers: prefill GPU, decode LPU, orchestration CPU. The KV cache, whose transfer is the bottleneck of disaggregation, could justify a dedicated storage tier, exactly what the cancelled Rubin CPX was meant to be. If KV transfers remain the limiting factor at scale, that tier will reappear, in one form or another, in the Feynman 2028 platform.

Sources and method

This article draws on a research file closed on May 24, 2026. Labels: verified fact = citable primary source; credible estimate = consistent but not independently verified; assumption = reasoning without measurement. A French version of this article, published on May 24, 2026, is the original.

Dynamo 1.0

NVIDIA Developer Blog, NVIDIA Dynamo 1.0 Production Ready: developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/, March 16, 2026 (verified fact). Multimodal E/P/D figures, Kubernetes Inference Gateway, FastVideo.
GitHub: ai-dynamo/dynamo, open source (verified fact).
GitHub: ai-dynamo/grove, Grove API for topology-aware scheduling on GB300 NVL72 (verified fact).
SemiAnalysis InferenceX, 7× throughput benchmark on GB200 NVL72 with DeepSeek R1-0528 FP4 (vendor-cited, relayed by NVIDIA).
Baseten, “How Baseten Achieved 2x Faster Inference with NVIDIA Dynamo”: baseten.co/blog/how-baseten-achieved-2x-faster-inference-with-nvidia-dynamo/, October 16, 2025, Abu Qader, Michael Feil, Rachel Rapp. TTFT -50%, +61% RPS, +62% TPS on Qwen3 Coder 480B (verified fact). The “2×” claim applies to TTFT. The “no additional hardware cost” claim comes from social media, not from the technical post (editorial nuance).

SGLang 0.5.12

GitHub release: sgl-project/sglang v0.5.12, May 16, 2026 (verified fact).
PD disaggregation with GPU Staging Buffer, Decode Radix Cache, NIXL/Mooncake/MORI-IO backends (verified fact).
HiSparse: 3× throughput at 256 concurrent requests, up to 5× on long contexts (verified fact, limited to DSA models: DeepSeek-V3.2, GLM-5.1).
HiCache: UnifiedRadixTree, SSD offload via Mooncake (verified fact).
DeepSeek V4 day-0, TP16 on H100/H20 (verified fact).

TensorRT-LLM 1.3

Releases: rc7 (March 10) to rc15 (May 21, 2026), pre-release (verified fact).
Helix Parallelism: arXiv 2507.07120, KV Parallelism (attention) + TP×EP (FFN), 1.5× TTL, 32× batch at constant latency on DeepSeek-R1/Blackwell (verified fact).
RocketKV: arXiv 2502.14051, ICML 2025 (NVlabs), 3× decode speedup, -31% peak memory on H100 (verified fact).
Service discovery (rc12), KV Cache Connector API (v1.1+), MQA/GQA Sparse Attention (rc13) (verified fact).

Tenstorrent Galaxy Blackhole

Tenstorrent Newsroom, GA April 28, 2026: tenstorrent.com/newsroom/ (verified fact).
Specs: 32 Blackhole chips, 23 PFLOPS Block FP8, 6.2 GB SRAM (2.9 PB/s), 1 TB DRAM (16 TB/s), 56× 800G Ethernet (verified fact).
Blitz Mode: 350+ tokens/s/user decode, TTFT under 4 s on 100k context, DeepSeek-R1-0528 671B, batch 8-64, context up to 128k (verified fact as vendor publication, not verified by independent benchmark).
Price: system starting at $110,000, 4-system supercluster starting at $440,000 (verified fact).

AMD

ROCm Blog, vLLM-ATOM plugin: rocm.blogs.amd.com, May 7, 2026 (verified fact).
MORI-IO (Modular RDMA Interface): github.com/ROCm/mori, RDMA library for KV and EP transfer on AMD GPUs (verified fact).
MI350P PCIe: AMD blog May 7, 2026, 144 GB HBM3E, 4 TB/s, 4.6 PFLOPS MXFP4, dual-slot PCIe Gen 5, 600 W TBP (verified fact).

MLPerf Inference v6.0

MLCommons, MLPerf Inference v6.0 Results: mlcommons.org/2026/04/mlperf-inference-v6-0-results/, April 1, 2026 (verified fact).
NVIDIA: 2.5M tokens/s DeepSeek-R1, 288 GPUs, 4× GB300 NVL72 (verified fact). The 2.7× is against the earlier NVIDIA submission, not against the whole v5.1 field (editorial clarification).
AMD MI355X: 100,282 tokens/s Llama 2 70B Server (3.1× vs MI325X), 1,042,110 tokens/s Offline / 1,016,380 Server on 11 nodes / 87 MI355X (verified fact).

Speculative decoding

EAGLE-3 / SpecForge: arXiv 2603.18567, March 2026. Measured range: 3.27× to 4.79× on HumanEval (LLaMA-3.3-70B-Instruct, temp=0) (verified fact).
P-EAGLE (AWS): March 13, 2026, main PR #32887. 1.55× at c=1, 1.05× at c=64 on MT-Bench/GPT-OSS-20B (verified fact).

Prices

All prices in this article are in US dollars, as reported by the cited sources (Tenstorrent’s published pricing, cloud GPU street rates). The French edition converts them to euros at an indicative 1 USD = 0.92 EUR (mid-2026).

The KV cache is no longer a side effect: it is the center of LLM serving in 2026

redaction@lecompute.fr (Killian Pluenet) — Fri, 03 Jul 2026 00:00:00 GMT

What changed since PagedAttention

The KV cache, the store of attention keys and values a model keeps so it never recomputes past tokens, used to be framed as a memory-consumption problem, and our French primer treated it that way: the cache grows linearly with context length and batch size, and it is the cache, not the weights, that saturates VRAM first. , the vLLM scheme that stores that cache in non-contiguous pages the way an operating system handles virtual memory, solved fragmentation; quantization (one byte per value instead of two) doubled capacity; prefix caching removed redundant recomputation. Those three levers remain the foundation. But each treats the KV cache as a local byproduct: a tensor the GPU owns, manages, and shares with no one.

What happened between late 2025 and May 2026 goes beyond that frame. The KV cache has acquired its own specialized quantization, distinct from the weights’. It now has a transfer protocol for migrating from one node to another. It lives in a four-tier memory hierarchy, from GPU (the high-bandwidth memory soldered next to the GPU) down to S3 object storage. And the inference scheduler makes its routing decisions based on the state of the cache on each node.

The KV cache is no longer a side effect. It is the central object around which the serving stack is reorganizing.

TurboQuant: the cache drops to ~3 bits

FP8 quantization of the KV cache (one byte per value instead of two) had been a production default since 2024. Going lower without hurting quality looked out of reach: the KV cache changes at every token, unlike the weights, which you calibrate once. Aggressive quantization schemes (4-bit, 2-bit) produced measurable artifacts on generation benchmarks.

TurboQuant, published at ICLR 2026 by Google Research and NYU (arXiv 2504.19874), solves this with two chained mechanisms.

The first, PolarQuant, applies a random orthogonal rotation to the K and V vectors before quantization. The idea is not to approximate the values as they are: it is to transform them into a space where their distribution becomes near-Gaussian, hence better suited to uniform quantization. The rotation is invertible and cheap: a randomly drawn orthogonal matrix, applied by multiplication, and undone after the attention computation. What would be expensive to store (the matrix) is handled by a fixed seed: the same matrix is rebuilt on every call, never kept.

The second, QJL (Quantized Johnson-Lindenstrauss), corrects the quantization residual with a 1-bit random projection. Instead of storing the error at full resolution, QJL projects it into a subspace and keeps only the sign of each projected component, one bit. The Johnson-Lindenstrauss lemma guarantees that the distance between vectors is preserved with high probability in the projected subspace, which is enough for the attention computation. The combined result: about 3 bits per coordinate for a quality indistinguishable from the baseline, 2.5 bits with marginal degradation.

The vLLM 0.20 integration has been live since April 15, 2026 (PR #38479). The CLI flag is --kv-cache-dtype turboquant_3bit_nc. A FlashAttention 3/4 extension for followed in PR #40092. The vLLM blog of May 11, 2026 published the first systematic study. This is no longer a research paper, it is a configuration parameter.

RocketKV: smart eviction

TurboQuant compresses every value in the cache. RocketKV (arXiv 2502.14051, NVlabs, ICML 2025) takes the problem from the other end: not storing the entries attention will never look at.

The mechanism runs in two stages. The first, SnapKV++, performs a coarse eviction: at regular intervals it evaluates which cache blocks contributed least to the recent attention score and drops them. It is a high-pass filter on temporal relevance: old, never-consulted blocks disappear. The second stage applies a hybrid top-k sparse attention over the surviving cache: instead of computing attention over every remaining vector, it keeps only the k most relevant entries, selected by a lightweight score.

Measured on H100: up to 3× speedup, 31% peak-memory reduction. Its integration into TensorRT-LLM as a sparse-attention backend makes it complementary to quantization: TurboQuant shrinks the size of each entry, RocketKV shrinks the number of entries.

FlashAttention 4: prefill comes down a step

The attention computation itself crossed a threshold in March 2026. FlashAttention 4 (arXiv 2603.05451, Zadouri, Dao et al., Together AI) reaches ~1,613 TFLOPS/s at 71% utilization on Blackwell (SM100+): the paper measures ~1.3× cuDNN and ~2.7× Triton on B200. The gain does not come from an “inference-only” mode: FA4 does have a backward pass and serves training too. It comes from an algorithm/kernel co-design: asymmetric pipelining cut for Blackwell, a faster exponential, better overlap of the MMAs (the Tensor Core matrix-multiply-accumulate operations).

vLLM 0.20 uses it as the default MLA backend for prefill on Blackwell (SM100+); on Hopper, FA3 fills that role. The link to the KV cache is direct: the faster the prefill, the less the cache-filling time weighs in the TTFT (time to first token), which makes prefix caching less critical on short sequences and even more valuable on long ones.

The KV cache memory hierarchy: KVBM G1→G4

Until 2025, the KV cache lived in HBM or did not exist. CPU offloading existed as a prototype (vLLM, LMCache), but with no tier management, no migration policy, no persistent storage. The cache was an ephemeral object, destroyed when the request ended.

NVIDIA Dynamo 1.0, generally available since March 16, 2026, changes that posture radically. Its KVBM (KV Block Manager) subsystem organizes the cache into a four-tier storage hierarchy that borrows its logic from CPU memory hierarchies (L1 / L2 / L3 / DRAM), but at datacenter scale.

G1 GPU HBM hot, ~µs

G2 Cross-node CPU RAM warm, ~ms

G3 / G3.5 Local SSDs + ICMS flash cold, ~10 ms

G4 S3 / Azure Blob archive, ~100 ms

Figure 1: The four KVBM tiers of Dynamo 1.0; the KV cache migrates from GPU HBM down to remote object storage according to its reuse frequency.

G1: GPU HBM. The hot tier, where the cache lives during the attention computation. Access latency on the order of a microsecond, bandwidth of several TB/s. This is the tier PagedAttention has managed since 2023. The difference is that KVBM treats it as one floor among four rather than the cache’s only living space.

G2: CPU RAM, potentially distributed. The cache of a finished or interrupted request migrates to host RAM instead of being destroyed. If a later request presents the same prefix, the cache is promoted back to G1 with no recomputation. The cost: one PCIe round trip, on the order of a millisecond for a typical block. Dynamo distributes G2 across several CPU nodes: a prefix’s cache is not confined to local RAM, it is reachable from any node in the cluster.

G3 / G3.5: local SSDs and ICMS. For caches whose reuse frequency no longer justifies RAM, KVBM steps down to flash storage. The standard G3 tier uses local or pooled NVMe. The G3.5 tier is specific to the Vera Rubin platform: ICMS (Inference Context Memory Storage), flash storage integrated into the BlueField-4 DPUs and driven by the DOCA Memos framework. ICMS is sized for the KV cache, not for generic storage, and offers a data path that bypasses the host CPU through the NIC.

G4: remote object storage. The archive tier: S3, Azure Blob Storage. A long-session cache (an agent’s context over several hours, the history of a RAG workflow) persists in G4 and reloads on demand. Access latency is measured in tens of milliseconds, but recomputing the prefix would have cost seconds. G4 offloading is part of the March 2026 Dynamo 1.0 release, not a feature bolted on afterward.

The migration policy between tiers is governed by the blocks’ reuse frequency. A block referenced once drops quickly to G3/G4; a block several requests share (system prompt, anchored RAG document) stays in G1 or G2. It is exactly the logic of a CPU cache with an extended LRU policy, applied to blocks of tens of megabytes instead of 64-byte lines.

SGLang: HiSparse and HiCache

SGLang 0.5.12 attacks the same hierarchy with two distinct systems, each targeting a segment of the problem.

HiSparse handles GPU → CPU offloading for models that support sparse attention, so far DeepSeek-V3.2 and GLM-5.1, which use the DSA (DeepSeek Sparse Attention) architecture. The mechanism is proactive: HiSparse identifies inactive KV cache entries before they saturate HBM and moves them to host memory, while keeping a hot buffer in HBM for high-access-frequency entries. Measured effect: over 3× the baseline throughput at 256 concurrent requests, up to 5× on long contexts. The first public description came in the LMSYS blog of April 10, 2026.

HiCache is a separate, broader system. It implements a UnifiedRadixTree with Sliding Window Attention support, which lets it manage fixed-attention-window models (DeepSeek V4 included) in a shared cache tree. SSD offloading goes through the Mooncake store, a distributed cache-transfer engine that also shows up in TensorRT-LLM. HiCache and HiSparse are complementary, not alternatives. HiSparse handles GPU↔CPU migration, HiCache handles the cache hierarchy extended down to SSD.

Transfer connectors: the KV cache in transit

In a disaggregated serving pipeline, where one node runs prefill and another runs decode, the KV cache has to travel. A 70-billion-parameter model with 8k tokens of context produces ~2.6 GB of KV cache in FP16; in (four bits per value), that is still hundreds of megabytes to move between nodes before decode can begin. Without an optimized protocol, this transfer cancels the gain of .

Three connectors emerged in parallel.

KV Cache Connector API (TensorRT-LLM). Introduced in TRT-LLM v1.1 (not v1.3, which refines the implementation), this API standardizes the cache’s serialization, transfer and restore operations. v1.3 rc12 adds shortcuts for lmcache and kvbm; rc15 introduces a cache_salt_id to identify cache blocks by signature rather than by address. The KV cache becomes an addressable object, not an anonymous buffer.

NIXL (vLLM). vLLM’s internal transfer connector, redesigned in version 0.21 (May 2026) to support bidirectional transfers between P (prefill) and D (decode) nodes. v0.21 also adds the MooncakeStoreConnector for distributed offloading, and DCP/PCP support in the offloading connector.

LMCache. An independent project that combines CacheGen (KV cache compression), CPU offloading, and distributed caching. Compatible with vLLM. The approach is complementary: LMCache compresses before the transfer, which cuts the inter-node bandwidth needed.

CacheBlend (EuroSys 2025) rounds out the picture with a selective reuse mechanism: instead of recomputing a RAG document’s KV cache on every request, CacheBlend restores the pre-existing parts and recomputes only the new tokens. Measured result: 2.2 to 3.3× TTFT reduction on RAG workloads.

KV-aware routing: the scheduler changes its criterion

All these storage and transfer mechanisms would be useless if the scheduler kept routing requests without regard for the state of the cache. This is the last piece of the puzzle, and the one that produces the most spectacular gains.

Dynamo 1.0’s Kubernetes Inference Gateway plugin embeds a token-aware routing algorithm. Instead of assigning a request to the least-loaded node (weighted round-robin, the norm), the router first checks which nodes already hold the KV cache for the incoming request’s prefix. If a node in G1 or G2 holds the cache for the system prompt or the RAG document, the request goes straight there: the prefix’s prefill is short-circuited, and the TTFT drops.

NVIDIA measures a 20× TTFT reduction and a 4× end-to-end latency reduction on workloads with recurring prefixes. Baseten, in a Dynamo deployment on Qwen3 Coder 480B with ~50k-token prompts, reports a 50% TTFT reduction, a 61% gain in requests per second, and a 62% gain in tokens per second. Their blog headline says “2x faster inference”: it is specifically the TTFT that is halved, not overall throughput.

Comparison: the KV cache by runtime

Feature	vLLM 0.20-0.21	SGLang 0.5.12	TRT-LLM v1.1-1.3	Dynamo 1.0
FP8 KV quantization	✓	✓	✓	✓ (via runtime)
TurboQuant ~3 bits	✓ (v0.20, PR #38479)	—	—	—
Sparse attention (RocketKV)	—	HiSparse (DSA)	✓ (sparse backend)	—
FlashAttention 4 (prefill)	✓ (SM90+, default)	—	—	—
Smart CPU offloading	✓ (reuse frequency)	✓ (HiSparse)	✓ (host offloading)	G2 (KVBM)
SSD offloading	—	✓ (HiCache + Mooncake)	—	G3 / G3.5 (KVBM)
Object-storage offloading	—	—	—	G4 (S3 / Azure Blob)
P/D transfer (disaggregation)	✓ (NIXL, bidirectional)	—	✓ (KV Cache Connector API)	✓ (KVBM + NIXL)
KV-aware routing	—	—	—	✓ (K8s Inference Gateway)
FlexKV (variable length)	✓ (v0.17.2+, Tencent TACO)	—	—	—
Automatic prefix caching	✓	✓	✓	✓

Table 1: KV cache features by runtime, May 2026. '✓' = available in production; 'β' = available in preview/experimental; '—' = absent.

If you are sizing your serving stack, three readings stand out. vLLM concentrates the quantization innovations (TurboQuant, FlexKV, FA4): it is the runtime where the cache is most compressed per entry. SGLang concentrates the local hierarchy innovations (HiSparse + HiCache): it is the runtime that pushes the cache furthest off the GPU, but for a subset of models. Dynamo brings coordination at the cluster level (KVBM + KV-aware routing) and plugs into vLLM or TRT-LLM as the underlying runtime. TRT-LLM provides the standardized connector API and the RocketKV backend, but delegates orchestration to Dynamo.

vLLM 0.21 and the full P/D pipeline

vLLM version 0.21 (May 15, 2026) marks the moment the KV cache becomes a first-class object in the disaggregated pipeline. The HMA (Heterogeneous Memory Architecture) integration with the offloading subsystem unifies the memory paths: a KV block follows a G1 → CPU → back-to-G1 trajectory with no format change. Bidirectional transfers between P and D nodes go through the redesigned NIXL connector: a decode node can now send cache back to a prefill node, not just receive it.

The MooncakeStoreConnector opens distributed offloading: a P node’s cache can transit through the Mooncake store to a remote D node without going through the P node’s host RAM. And DCP/PCP support in the offloading connector lays the ground for architectures where the prefill and decode nodes are not on the same physical network.

The TOKENSPEED_MLA attention backend, optimized for Blackwell GPUs, rounds out the picture by accelerating the MLA computation (Multi-Latent Attention, DeepSeek V3/V4’s attention architecture) on the newest hardware.

Multimodal extends the hierarchy

Dynamo 1.0 is not limited to text. Its multimodal E/P/D (Embedding / Prefill / Decode) pipeline adds a third cache tier: the embedding cache for precomputed visual representations. On Qwen3-VL-30B-A3B-Instruct in FP8 on GB200, NVIDIA measures a 30% TTFT reduction and 25% higher throughput on image requests, a gain that comes entirely from the fact that the visual embedding, expensive to compute, is computed only once.

Video follows the same logic. FastVideo + SGLang Diffusion generates a 5-second 1080p video in ~4.5 s on a single B200, a throughput that makes real time plausible for the first time on standard datacenter hardware.

What comes next: BlueField-4 and the cache as network infrastructure

The Vera Rubin platform writes the KV cache into network silicon. The BlueField-4 DPU, announced at CES in January 2026 and expected in the second half of 2026, embeds DOCA Memos, a framework for KV cache communication and storage directly in the DPU. The G3.5 tier (ICMS) is not an SSD the DPU addresses: it is flash storage built into the network card, reachable without crossing the host CPU or the main PCIe bus.

There the KV cache is treated as a network object, not as application data. The DPU can route it, replicate it, compress it on the data path, the same way a smart network switch manipulates packets without the CPU getting involved. This is the next frontier: the cache is no longer only managed by the runtime, it is transported by the network infrastructure itself.

Silicon is reorganizing around the cache too, and not only at NVIDIA: Google’s TPU 8i tripled its on-chip SRAM (384 MB) precisely to house a larger KV cache closer to the compute.

MLPerf v6.0: the cache in the official results

The MLPerf Inference v6.0 round (April 1, 2026) brings an indirect but significant confirmation. NVIDIA’s submissions on B200 use the FP8 KV cache as the reference configuration, not FP16, not FP32. It is the first time a reduced cache format appears in official benchmark submissions as a baseline, not as an optional optimization. AMD, for its part, uses a pre-shuffled KV layout, a pre-optimized memory arrangement of the cache to reduce HBM bank conflicts. Even among the challengers, the cache has become a first-order optimization axis.

Conclusion

The KV cache’s trajectory in 2026 follows the one CPU cache memory traced in the 1990s: a performance byproduct turned full-fledged subsystem, with its own hierarchy, its own policies, and its own communication primitives. The next steps are visible. AMD’s pre-shuffled KV layout and LMCache’s CacheGen compression point toward a convergence on standardized cache formats, a GGUF of the KV cache portable across runtimes. DOCA Memos and BlueField-4 write the cache into the DPU, which moves the management boundary from the runtime toward the network. And Dynamo’s KV-aware routing raises the next question: if the scheduler knows the state of every node’s cache, is the next step not a market for the cache, where a node that has invested compute in a prefix sells it to the nodes that need it, instead of letting it die?

Sources and method

TurboQuant. Zandieh et al. (Google Research), ICLR 2026, arXiv 2504.19874. vLLM integration: PR #38479 (April 15, 2026), PR #40092 (FA3/FA4 extension). vLLM blog “First Comprehensive Study of TurboQuant” (May 11, 2026). The 6× memory reduction and 8× attention speedup are the paper’s claims, FP32 baseline; verified fact (paper + vLLM reproduction).

RocketKV. Zhang et al., ICML 2025, arXiv 2502.14051. 3× decode speedup and 31% peak-memory reduction on H100; verified fact (paper). TRT-LLM integration as sparse backend; verified fact (TRT-LLM release notes).

FlashAttention 4. Zadouri, Dao et al., arXiv 2603.05451, March 2026 (“Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling”). ~1,613 TFLOPS/s, 71% utilization on B200; ~1.3× cuDNN, ~2.7× Triton; verified fact (paper). FA4 targets Blackwell (SM100+) and has a backward pass (training). MLA prefill backend of vLLM 0.20 on Blackwell; verified fact (vLLM 0.20 release notes).

vLLM 0.20 / 0.21. Release notes: vLLM 0.20.0 (April 27, 2026), vLLM 0.21.0 (May 15, 2026). FlexKV (PR #34328, Tencent TACO) shipped since v0.17.2, not introduced in v0.20; verified fact. Smart CPU offloading (PR #35342) appeared in v0.18.0; verified fact.

SGLang 0.5.12: HiSparse and HiCache. LMSYS blog, April 10, 2026: HiSparse. 3-5× throughput at 256 concurrent requests (DSA workloads); verified fact (LMSYS blog). HiSparse and HiCache are distinct systems, not interchangeable; verified fact (SGLang 0.5.12 release notes).

NVIDIA Dynamo 1.0: KVBM. NVIDIA Developer blog: Dynamo 1.0 Production Ready (March 16, 2026). G1-G4 tiers, S3/Azure Blob offloading; verified fact (official blog). Token-aware routing: 20× TTFT, 4× e2e latency; NVIDIA figures on recurring-prefix workloads, to be read as an “up to” bound (not independently reproduced).

Baseten. Blog: How Baseten Achieved 2x Faster Inference with NVIDIA Dynamo. 50% TTFT reduction, 61% RPS, 62% TPS on Qwen3 Coder 480B (~50k-token input). The “2×” is specifically the TTFT; verified fact (Baseten blog, May 2026).

TensorRT-LLM: KV Cache Connector API. Introduced in TRT-LLM v1.1, not v1.3; verified fact (v1.1 release notes). v1.3 rc12: lmcache/kvbm shortcuts; rc15: cache_salt_id; verified fact (GitHub changelogs).

BlueField-4 and DOCA Memos. CES January 2026 announcement. ICMS (Inference Context Memory Storage) and DOCA Memos; verified fact (NVIDIA CES 2026 press release). H2 2026 availability; verified fact (official roadmap).

CacheBlend. EuroSys 2025. 2.2-3.3× TTFT reduction on RAG workloads; verified fact (EuroSys paper).

MLPerf v6.0. April 1, 2026. FP8 KV cache in NVIDIA B200 submissions, AMD pre-shuffled KV layout; verified fact (public MLPerf results).

Multimodal Dynamo. +30% TTFT, +25% throughput on Qwen3-VL-30B/GB200 (image requests); verified fact (NVIDIA Dynamo blog). FastVideo 1080p 5s in ~4.5s on a B200; verified fact (NVIDIA demo).

This article is the English adaptation of a French original, published on May 20, 2026.

How much VRAM do you need to run an LLM locally?

redaction@lecompute.fr (Killian Pluenet) — Fri, 03 Jul 2026 00:00:00 GMT

The 2 GB per billion parameters rule

A LLM parameter is nothing more than a number. The only question that sets its memory cost is: how many bytes do you store it on? Models are trained and shipped in 16-bit, the format or its cousin BF16, so 2 bytes per parameter. The arithmetic is then direct: a 7-billion-parameter model weighs 7 × 2 = 14 billion bytes, or ~13 GiB of weights. That is all the famous “2 GB per billion” rule says: it is not an empirical approximation, it is the arithmetic of the format.

What makes the rule useful is that it carries over to any precision. Drop the weights to 8-bit and it is one byte per parameter, half. In 4-bit it is half a byte, a quarter. does not change the number of parameters, only the number of bits that encode each one: it is a multiplication, not a rewrite of the model.

Three line items, not one

The VRAM consumed by a model that serves is not that of the weights file alone. Three line items add up, and it is their sum that must fit on the card:

The weights: fixed, given by the rule above. It is the only line item most people compute, which is why many get it wrong.
The KV cache: the memory of what the model has already read. It starts at zero and grows with every token generated, proportionally to the context length and to the number of requests served in parallel. It is the KV cache, not the weights, that saturates VRAM first on long contexts.
The overhead: the context, activation buffers, allocator fragmentation. Budget ~1 GB incompressible, plus a few percent of the weight volume.

This breakdown explains the classic trap: a model that “fits” in VRAM at load time overflows during use. The weights fit; the KV cache of a long context had not been budgeted.

The table: what the weights weigh

Before adding the rest, you need to know what the bare model costs. The table below applies the rule to the three most common sizes, in the three deployment precisions:

Model	16-bit (FP16)	8-bit (INT8)	4-bit
7–8 B	≈ 14–16 GB	≈ 7–8 GB	≈ 4–5 GB
13 B	≈ 26 GB	≈ 13 GB	≈ 8 GB
70 B	≈ 140 GB	≈ 70 GB	≈ 42 GB

Table 1: VRAM of weights alone, by model size and precision. 16-bit = 2 bytes/parameter; 8-bit = 1 byte (exact arithmetic). 4-bit = real Q4_K_M: ~4.8 bits/weight, or ~0.6 byte, a bit more than the theoretical quarter, because of the per-block scale factors. Values rounded to the order of magnitude.

The 4-bit column deserves a caveat: a quantized format never drops exactly to half a byte per parameter. Schemes like Q4_K_M (the quantized weight file that llama.cpp loads) store, on top of the weights, per-block scale factors covering a handful of values: the real footprint lands around 4.8 bits per parameter, or ~0.6 byte. That is why a 70B “in 4-bit” (Q4_K_M) weighs ~42 GB and not 35: the difference is the quantization metadata, and it is not optional.

Which card for which model

In practice, available VRAM comes in tiers, those of the cards on the market. Setting the tiers against the models gives a direct decision grid:

VRAM	Typical cards	What fits comfortably
8 GB	RTX 4060, entry-level cards	7–8 B in 4-bit
12–16 GB	RTX 4070, RTX 4060 Ti 16G	13 B in 4-bit, 7–8 B in 8-bit
24 GB	RTX 3090, RTX 4090	32 B in 4-bit, 13 B in 8-bit
32 GB	RTX 5090	32 B in 4-bit (comfortable); 70 B only in ~3-bit (very tight)
48 GB	RTX A6000, 2× 24 GB	70 B in 4-bit
80 GB	H100, A100 80G	70 B in 8-bit (moderate context) or 4-bit (long context)

Table 2: What fits comfortably per VRAM tier, leaving margin for the KV cache and overhead. 'Comfortable' = weights + usual context; 'tight' = weights only, short context.

Two readings of this table are worth spelling out. First: the price jump between 24 and 80 GB does not reflect a proportional jump in raw capacity, but access to 70B models and long contexts, which is the whole stake of the RTX 5090 versus H100 matchup for local LLMs. Second: past 24 GB, the trade-off is no longer just “buy a bigger card” but “buy or rent”: a 70B served a few hours a day costs less in cloud GPU than in amortized hardware.

Reducing the VRAM you need

When the model you want does not fit, three levers exist, in decreasing order of payoff.

The first, and by far the most powerful, is weight quantization. Going from 16 to 4-bit divides the main line item by four, which fits a 70B in 4-bit (~42 GB) onto two consumer cards, where its 140 GB in FP16 force several datacenter GPUs. The quality loss of a well-calibrated Q4_K_M stays small on most workloads; it is the best-established trade-off in local inference.

The second is KV cache management. Reducing the maximum context length, quantizing the cache to (one byte per value instead of two, so double the capacity), or enabling prefix caching frees exactly the VRAM that long contexts were claiming. On a high-context workload, this lever returns more usable memory than a card upgrade.

The third is offloading: moving part of the layers to CPU RAM, as llama.cpp allows. It unblocks running a model too big for VRAM, but at a steep price: every layer in RAM is paid for in round trips over the bus, and throughput collapses. It is a last-resort fix, not a sizing strategy.

Conclusion

The question “how much VRAM for an LLM?” has an arithmetic answer for the weights, two gigabytes per billion divided by the precision, but the real sizing question is elsewhere. It is not “does the model fit?”, it is: what context length and what level of concurrency do I have to absorb, and how much VRAM is left to house them once the weights are down?

Sources and method

The 2 bytes per parameter in 16-bit rule is a fact: it follows from the IEEE 754 half-precision (FP16) and bfloat16 formats, both encoded on 16 bits. The overhead of 4-bit formats (~4.8 effective bits per parameter for Q4_K_M, or ~42.5 GB for a 70B) is measured on the GGUF k-quant schemes: see the llama.cpp quantization documentation and the GGUF file sizes published on Hugging Face. The Table 1 sizes are orders of magnitude computed from the rule; exact values vary by a few percent from one model to another (shared heads, tied embeddings). The Table 2 tiers rest on the published VRAM capacities of the cards cited (RTX 4060/4070/4090/5090, RTX 3090, A6000, A100/H100); “comfortable” is an estimate incorporating a usual margin for the KV cache and overhead, not a guarantee: the real context and batch decide. The sizing of the KV cache and overhead is detailed in a dedicated article. A French version of this article, published June 1, 2026, is the original.

Image credit. Header photo: NVIDIA GeForce GTX 780 PCB by GBPublic_PR, CC BY 2.0, via Wikimedia Commons, cropped to the die and its ring of memory chips.

RTX Spark vs DGX Spark: two different chips, one 273 GB/s wall

redaction@lecompute.fr (Killian Pluenet) — Fri, 03 Jul 2026 00:00:00 GMT

The RTX Spark is not a rebadged DGX Spark

The press coverage of May 31, 2026 immediately blended two products that NVIDIA sells separately, and the conflation deserves to be untangled before any technical analysis. The DGX Spark has existed since 2025: a Linux (DGX OS) mini AI development workstation built on the GB10 superchip, aimed at the data scientist who wants a Grace Blackwell node on their desk. The RTX Spark announced at Computex is something else: a consumer SoC, codename N1X, that runs Windows on Arm and targets creators, developers and gamers.

The two share the philosophy (an Arm CPU and a Blackwell GPU on the same package, 128 GB of unified memory), and Jensen Huang himself described the N1X as technically very close to the GB10 in a press Q&A. But close is not identical. These are two chips, two operating systems, two form factors, two product sheets, and probably two distinct sets of networking features. Reading the RTX Spark as a DGX Spark under another name misses what NVIDIA is actually doing: entering the premium Windows-on-Arm PC market for the first time, where Qualcomm had so far been alone.

What makes the analysis possible despite NVIDIA’s near-silence on detailed specs is precisely that kinship. The DGX Spark’s GB10 is documented, measured, shipping; the N1X is not yet. The whole exercise is therefore to derive from the twin what NVIDIA has not published about the N1X, flagging at every step what is official, what comes from the GB10, and what is leak or estimate.

The silicon: Grace Blackwell for Windows

The N1X fuses an Arm CPU and a Blackwell GPU on a single package, connected by a coherent link. Each block has its own lineage.

Block	RTX Spark (N1X)	Where the figure comes from
CPU	20 Arm cores: 10× Cortex-X925 + 10× Cortex-A725, co-designed with MediaTek	Leak / reporting
GPU	Blackwell "RTX", 6,144 CUDA cores (≈ RTX 5070)	Official (press release)
Tensor / RT Cores	5th-gen Tensor Cores (FP4/NVFP4), 4th-gen RT Cores	Official + GB10
Compute capability	SM121 (≠ SM120 on the RTX 5090, ≠ SM100 datacenter)	GB10 (Backend.AI)
Process	TSMC 3 nm ("3nm-class")	Official
Memory	up to 128 GB unified LPDDR5X, 256-bit bus, ~273 GB/s	Official (128 GB) + GB10 (273 GB/s)
CPU↔GPU link	NVLink-C2C, ~600 GB/s aggregate	Hot Chips 2025
OS	Windows on Arm	Official
TDP	~45–110 W depending on the OEM (power supplies up to 140 W)	Unofficial

Table 1: RTX Spark (N1X) architecture, best-reported as of June 5, 2026. GPU and memory figures are derived from the twin GB10; the core split comes from heise/c't reporting and leaks, not from an official NVIDIA spec sheet.

The CPU is a 20-core Arm design (ten Cortex-X925 performance cores, ten Cortex-A725 efficiency cores), co-designed with MediaTek, which supplies the CPU chiplets. The X925 are the same cores as in the Dimensity 9400 smartphone chip: NVIDIA did not design a custom server CPU here, the way it did with the Olympus core of its Vera CPU. It assembled a platform out of leading-edge Arm IP.

The GPU is an “RTX” Blackwell with 6,144 cores (the same count as the desktop GeForce RTX 5070), with 5th-generation that wire into silicon and 4th-generation RT Cores for rendering. On the twin GB10, NVIDIA configures this GPU as 48 / 192 Tensor Cores. This GPU is what justifies the “RTX” badge and the gaming promise.

One engineering nuance matters to anyone writing kernels. The Spark’s Blackwell is not the datacenter Blackwell. The GB10 (and therefore the N1X) is compute capability SM121, distinct from the SM120 of an RTX 5090 and the SM100 of a B200. It supports FP4/FP6/FP8 and 5th-gen Tensor Cores, but its Tensor Core programming model is closer to Ampere’s mma.sync than to the tcgen05 instructions of datacenter Blackwell: it lacks TMEM and 2-SM cooperative MMAs. Concretely, a low-level CUDA kernel developer targeting all three now maintains three code paths (Hopper, datacenter Blackwell, consumer Blackwell) instead of two. For users of runtimes like vLLM, TensorRT-LLM or llama.cpp, the abstraction layer absorbs the difference; for whoever writes the kernel, it is real.

Three numbers people conflate

This is where mainstream coverage slips the most, and where precision matters. Three values circulate around the RTX Spark, expressed in the same unit, and the press swaps them freely even though they describe three different things.

Metric	Value	What it is
LPDDR5X memory bandwidth	~273 GB/s	Unified system memory throughput (the decode wall)
Memory bus	256-bit (N1X) / 128-bit (N1)	Channel width, not a throughput
NVLink-C2C	~600 GB/s aggregate	Coherent CPU↔GPU link, not memory

Table 2: Three bandwidths not to confuse. The only one that sets generation speed is the first.

The memory bandwidth is ~273 GB/s. NVIDIA has published no official figure for the N1X; this one comes word for word from the GB10 datasheet: “Memory Bandwidth: 273 GB/s · Memory Channels: 16 channels (256 bit) LPDDR5X 8533”. The arithmetic checks out: a 256-bit bus moves 32 bytes per transfer, and at 8,533 MT/s that gives 32 × 8,533 ≈ 273 GB/s. The “up to 300 GB/s” seen elsewhere is a rounding of a 9.4 Gbps clock mentioned at Hot Chips 2025; the shipping, measured value is 273.

The ~600 GB/s figure is something else: the aggregate bandwidth of NVLink-C2C, the coherent link that connects the CPU die to the GPU die inside the package. NVIDIA described it at Hot Chips 2025 as offering “5x the bandwidth of PCIe Gen 5”: it is an internal interconnect, not a throughput to system memory. Confusing it with memory bandwidth is like confusing the width of a datacenter node’s with the throughput of a card: two different stages of the transport hierarchy.

This memory is unified and coherent: the GPU has no dedicated VRAM, it reaches the LPDDR5X through memory controllers that live in the MediaTek CPU die, transparently over NVLink-C2C. It is the same architectural principle as Apple Silicon, or as a Jetson Thor at the embedded end of the spectrum: a single pool that CPU and GPU share dynamically, with no explicit copies. The benefit is capacity: 128 GB visible to the GPU, where a consumer card tops out at 32. The cost is throughput: LPDDR5X is mobile memory, not GDDR7 and not HBM.

The “1 petaflop”, decoded

The announcement’s headline number (1 petaflop) is accurate and misleading at the same time, and it needs unfolding like any raw figure. NVIDIA’s official footnote is explicit: “Theoretical FP4 TOPS using the sparsity feature”. Three words do all the work there.

Theoretical: it is a peak of the compute units, not a sustained throughput on a real workload. FP4: it is the densest format of the 5th-gen Tensor Cores, four bits per value, not FP16. Sparsity: it is structured 2:4 , where the hardware skips the half of the weights that were zeroed out in advance, doubling the advertised rate without computing more. Remove the sparsity and you get ~500 dense FP4 TFLOPS, half the shop-window figure. Move up to FP16 or FP32, formats in which plenty of training and serving still happens, and the throughput falls back to that of a mid-range GPU, RTX 5060/5070 class.

That FP4 peak is not useless: it serves , the phase where the model processes the whole prompt at once and saturates the Tensor Cores. But a real inference service is almost always dominated by the next phase, , and that one does not read the FLOPS spec sheet.

What ~273 GB/s means for inference

Here is the passage that decides everything else. LLM token generation is autoregressive: every new token re-reads the entirety of the model’s weights plus all the accumulated so far. For a few billion useful operations, the GPU traverses tens of gigabytes of memory. The workload is memory-bound: the silicon spends most of its time waiting on memory, and bandwidth, not FLOPS, sets the ceiling.

Run the arithmetic on a dense case. A 70-billion-parameter model served in weighs ~70 GB of weights, which must be re-read for every token. The physical floor of generation is a division: 70 GB ÷ 273 GB/s ≈ 256 ms per token, which is ~4 tokens per second at the memory ceiling, for a single sequence. An RTX 5090, with its ~1,792 GB/s of GDDR7, would read the same weights in ~39 ms, but it cannot hold a 70B FP8 model: its 32 GB overflow. That is the Spark’s whole paradox: it loads the model the 5090 refuses, then serves it at whatever pace its mobile memory allows. The same mobile memory that provides the 128 GB imposes the 273 GB/s; you do not get one without the other.

The escape hatch is not a trick, it is a model type. MoE architectures (Mixture of Experts, where a router activates only a few experts per token out of all those available) break the equation. On a GPT-OSS 120B, only ~4 experts out of 128 fire per token: the full model (~65 GB of weights, ~80 GB loaded) must reside in memory, but each token only reads a fraction of it. The memory cost per token collapses, and decode throughput becomes usable again even though the model weighs 80 GB. That is precisely what the 128 GB are for: load the whole model, then pay in bandwidth only for the active experts. For a dense model, the same capacity merely houses the model; it does not make it run fast.

The only measured numbers available come from the DGX Spark under Linux, to be used as a proxy while keeping in mind that the N1X’s Windows-on-Arm performance may differ (drivers were reported immature in early 2026). The pattern there is unambiguous.

Workload	Prefill (tps)	Decode (tps)
GPT-OSS 20B (MXFP4, Ollama) on DGX Spark	≈ 2,053	≈ 50
GPT-OSS 20B on RTX 5090 (ref.)	≈ 8,519	≈ 205
GPT-OSS 120B (MXFP4, llama.cpp) on DGX Spark	≈ 1,723	≈ 38
GPT-OSS 120B on a trio of RTX 3090s (ref.)	≈ 1,642	≈ 124

Table 3: Measured throughputs on the DGX Spark (GB10, Linux), indicative for the N1X. Prefill = prompt processing (compute-bound); decode = generation (memory-bound). Sources: LMSYS (Oct 13, 2025) and community benchmarks.

The DGX Spark holds its own at prefill: it matches a trio of RTX 3090s on GPT-OSS 120B, because prefill is compute-bound and the FP4 peak earns its keep there. But on decode it generates three to four times more slowly than GDDR-equipped cards: the 5090 and the 3090 trio have the bandwidth, the Spark has the capacity. LMSYS closes its review with a sentence that sums up the product: “the Spark’s unified LPDDR5x memory bandwidth is the main limiting factor”. The RTX Spark inherits that memory, and that limit, unchanged.

One software lever remains, specific to the NVIDIA ecosystem: the FP4 formats. NVFP4 (NVIDIA’s variant with 16-value blocks and two-level scaling, finer-grained than the standard MXFP4) is wired into the Spark’s Tensor Cores, but it is only exploited in hardware through TensorRT-LLM or NIM; a file in llama.cpp uses its own scheme. Aggressively quantizing the weights reduces the bytes read per token, so it nudges the memory-bound ceiling upward, but it does not remove it. helps too: serving several concurrent requests amortizes the weight re-read across the whole batch. The counter to the 273 GB/s wall comes down to three moves (MoE models, NVFP4/MXFP4 quantization, batching): none of them changes the memory, all of them route around its slowness.

The RTX Spark against the rest of the market

The right question is not “is this a good GPU” but “what capacity/bandwidth trade-off is this”. Set against the machines a buyer actually compares, the Spark occupies a precise slot.

System	Memory	Memory BW	Indicative price
RTX Spark (N1X)	128 GB unified LPDDR5X	~273 GB/s	≈ $2,899 (est.)
DGX Spark (GB10)	128 GB unified LPDDR5X	~273 GB/s	$3,999 → $4,699
Mac Studio M3 Ultra	96 GB unified (256/512 GB pulled)	~819 GB/s	from $5,299
RTX 5090 (desktop)	32 GB GDDR7	~1,792 GB/s	$1,999 MSRP, $2,999+ street
RTX PRO 6000 Blackwell	96 GB GDDR7 ECC	~1,792 GB/s	$13,250 (official list)

Table 4: RTX Spark against competing machines. Prices in US dollars: MSRPs or analyst estimates (see Sources and method). FP4 values are vendor sparse peaks.

Against the DGX Spark, it is the same capacity/bandwidth couple. The differences are the OS (Windows vs Linux), the form factor (consumer PC vs mini workstation), the lower estimated price, and probably the absence of the ConnectX-7 clustering that the DGX Spark exposes to pair two units. If what you need is a Linux AI dev node, the DGX Spark remains the product; if it is a premium Windows PC that can also do local inference, that is the RTX Spark.

Against the Mac Studio M3 Ultra, the trade-off shifted mid-shortage. Apple offers three times the bandwidth (~819 vs ~273 GB/s), hence visibly faster token generation, but pulled the 256 and 512 GB options from its catalog in spring 2026 and raised the base to $5,299 for a 96 GB cap: the capacity argument switched sides. But no hardware FP4 and no CUDA. The Spark wins at prefill and on the CUDA/TensorRT ecosystem; the Mac wins at decode and on capacity. A community trick combines both in a disaggregated serving setup (DGX Spark for prefill, Mac Studio for decode) for a ~2.8× gain over the Mac alone.

Against an RTX 5090 or an RTX PRO 6000, it is exactly the trade-off that already structures the consumer-versus-datacenter matchup. The GDDR7 cards have ~6.5× the Spark’s bandwidth, hence ~4× its generation throughput on models that fit in their VRAM. But 32 GB (5090) or 96 GB (6000 Pro) cap the model size. The Spark trades raw speed for capacity: it loads a 120B or a 70B that a 5090 cannot, at the price of a lower token rate. To know precisely what fits in each memory tier, you apply the full formula: weights, KV cache and overhead. The head-to-head with the 5090, verdicts by model size included, is in DGX Spark vs RTX 5090.

Price, availability, and what remains unconfirmed

No price is official. The most-cited estimates come from a Morgan Stanley report relayed on June 2, 2026: N1X-based AI PCs would need to sell around $2,899 (base configurations), N1 models around $1,799, figures that concern entry configurations (16–32 GB of memory, 512 GB to 1 TB of SSD). A 128 GB, multi-terabyte machine will cost substantially more. PCWorld cites OEM sources around $2,500 (N1X) and $2,000 (N1). The explicit target is the MacBook Pro M5 Pro, at around $2,100.

Availability is announced for fall 2026, with a slippage risk tied to the Windows-on-Arm drivers reported immature in early 2026. Every laptop shown at Computex was a non-functional mockup; only NVIDIA’s internal systems were running. More than 30 laptops and ~10 desktops are expected from ASUS, Dell, HP, Lenovo, Microsoft Surface and MSI, with Acer and Gigabyte to follow.

Who it is for, and what for

The decision reads off the shape of your workload, not off the spec sheet.

To load a large model locally (a 100 to 120B MoE that no consumer card can hold), the RTX Spark is exactly the tool: 128 GB of unified memory, and a low memory cost per token if the model is MoE. For a Windows creator PC that must also handle inference and CUDA-accelerated rendering, it is the first Windows-on-Arm machine to offer 100% of the NVIDIA stack. To generate fast on a dense model (a 70B served at quality at several dozen tokens/s), it is the wrong machine: its mobile memory rules that out by construction, and a GDDR card or a higher-bandwidth Mac Studio stays ahead. For a pure Linux AI dev node, the twin DGX Spark, or renting a cloud GPU by the hour, remains the rational option.

Conclusion

The RTX Spark does not move the fundamental limit it shares with the DGX Spark: same memory type, same bus, same ~273 GB/s. What it brings is capacity and the energy efficiency of an SoC, not token throughput on dense models. The real question is therefore not “how many petaflops” but the one every inference machine purchase poses: how much bandwidth to re-read your weights on every token, and how much memory to house them once loaded? The Spark answers the second loudly and the first modestly.

What will move that wall is not a better GPU, it is better memory. NVIDIA has publicly committed to three generations: Grace Blackwell today, then a Vera Rubin pairing on LPDDR6, then Rosa Feynman. LPDDR6 is precisely the lever that would lift a consumer SoC past 273 GB/s, and the same phase-specialization logic is already reshaping datacenter inference. The 2026 RTX Spark is the first step: a bet on unified capacity, while mobile bandwidth catches up.

Sources and method

This article draws on an internal research file closed on June 5, 2026, the RTX Spark having been announced on May 31 / June 1, 2026 at Computex/GTC Taipei. Since NVIDIA has published almost no detailed official N1X specs, part of the figures are derived from the twin GB10 (DGX Spark) or from leaks; each label below says which is which. A French version of this article, published on June 5, 2026, is the original.

Verified facts. The NVIDIA newsroom release gives the GPU at 6,144 CUDA cores, the “1 petaflop”, the 128 GB of unified memory, the 20-core CPU and Windows on Arm as the OS. The ~273 GB/s memory bandwidth, the 256-bit bus (16 LPDDR5X-8533 channels) and the SM121 compute capability are facts verified on the twin GB10 (DGX Spark datasheet; Backend.AI analysis, February 2026), not on the N1X: NVIDIA has published no bandwidth figure for the RTX Spark. The ~600 GB/s aggregate NVLink-C2C and the “5x PCIe Gen 5” ratio come from the Hot Chips 2025 presentation. The line “Theoretical FP4 TOPS using the sparsity feature” is NVIDIA’s official footnote; the ~500 dense FP4 TFLOPS follows from it (2:4 sparsity = ×2).

Credible estimates. The prices ($2,899 / $1,799 for N1X / N1) are Morgan Stanley analyst estimates relayed by Max Weinbach on June 2, 2026, unconfirmed by NVIDIA, against a DRAM/NAND shortage that has already pushed the DGX Spark from $3,999 to $4,699. PCWorld’s OEM sources (≈ $2,500 / $2,000) converge to the same order of magnitude. The core split (10+10, 9+9, 8+4), the 256/128-bit buses and the 45 to 110 W TDP are best-reported (heise/c’t, VideoCardz leaks), not official.

Measurement proxy. All tokens/s figures (GPT-OSS 20B and 120B, prefill/decode) are measured on the DGX Spark under Linux, not on the RTX Spark under Windows: LMSYS review of October 13, 2025, community benchmarks (Benjamin Marie, The Kaitchup; the llama.cpp #16578 discussion). The N1X’s Windows-on-Arm performance may differ (drivers reported immature). The RTX 5090 / RTX 3090 references are third-party measurements on the same workloads.

Stated assumptions. The ~4 tokens/s physical floor on a dense 70B FP8 model (70 GB ÷ 273 GB/s) is a memory-ceiling calculation for a single sequence, not an application throughput. The MoE reasoning (bytes read per token reduced by expert routing) is a mechanism, not a measurement on the N1X. The LPDDR6/Vera Rubin/Feynman projection rests on NVIDIA’s public roadmap, with no detailed timeline.

Primary sources. NVIDIA Newsroom release (Computex 2026 announcement); DGX Spark / GB10 datasheet; NVIDIA Developer Blog: Introducing NVFP4; TensorRT-LLM documentation (DGX Spark target); Apple Mac Studio M3 Ultra specs for the 819 GB/s.

Prices. All prices in this article are in US dollars: manufacturer MSRPs or analyst estimates as reported by the cited sources. The RTX 5090 street price ($2,999+ on the US secondary market against a $1,999 MSRP) and the RTX PRO 6000 official list price ($13,250 in July 2026, up from ~$8,565 at launch) reflect the DRAM shortage inflating the whole memory-heavy segment; the same shortage made Apple pull the Mac Studio’s 256/512 GB options (spring 2026, Apple store + press coverage) and raise the M3 Ultra base to $5,299. European street prices differ (VAT, local margins); the French edition quotes euro conversions at an indicative 1 USD = 0.92 EUR.

Image credit. Header photo: NVIDIA DGX Spark by Daniel Lu, CC BY-SA 4.0, via Wikimedia Commons.

TorchTPU, XLA, JAX: how Google attacks NVIDIA's software lock-in

redaction@lecompute.fr (Killian Pluenet) — Fri, 03 Jul 2026 00:00:00 GMT

Why the real 2026 battle is not a FLOPS war

AI compute gets endless commentary, but its software layers rarely get explained. People talk about the chips (Ironwood, Blackwell, Trainium) as if the silicon were self-sufficient. The playing field has shifted: what decides an accelerator’s adoption today is no longer its peak , it is the friction a developer hits getting their existing PyTorch code to run on it.

This friction has a name and an address. The default path of AI development runs through : the code stack, the kernels, the tooling, the muscle memory of debugging at 2 a.m., all of it native to NVIDIA. It is a software moat, not a hardware one, and it is exactly what Google set out to dismantle. The 2026 shift is that Google stopped answering only with silicon and started answering with a stack: a framework developers love (PyTorch), a compiler that acts as the pivot ( ), a serving engine (vLLM) and, at the top, a project named TorchTPU whose ambition fits in one line from the team: “it should feel like PyTorch”.

This piece takes the problem by that friction. It speaks to two readers at once: the ML engineer who wants the detail (execution model, compilation path, install commands, benchmark numbers) and the reader who wants the whole picture: who depends on whom, and why it reshuffles the deck against NVIDIA. Each brick is explained first for what it does, then for the place it holds in the edifice. We start with the keystone, because nothing stands without it: the compiler.

XLA: the compiler that pivots the whole edifice

XLA (Accelerated Linear Algebra) is a compiler for machine learning. To see what it changes, you have to see how a framework runs without it. PyTorch, by default, handles each operation one after another: a kernel for the add, a kernel for the multiply, a kernel for the reduction. Each kernel reads its inputs from memory, computes, and writes its result back to memory, where the next kernel will read it again. On an accelerator, these memory round-trips cost far more than the computation itself.

XLA does not receive an operation: it receives the whole compute graph, and rewrites it for the target machine. Its signature transformation is called fusion. On the canonical example from the OpenXLA documentation (a computation of the form reduce_sum(x + y * z)), three separate kernels launch without compilation: one for the multiply, one for the add, one for the reduction. XLA melts them into a single kernel, and that is the detail that decides everything: it never writes the intermediate values (y*z, then x + y*z) to memory. It makes them “stream” straight from one step to the next, keeping them in the GPU registers. The documentation is explicit and worth quoting word for word:

XLA’s operation comes down to a chain. A framework hands it a graph expressed in , the versioned operation set that serves as the portability layer between frameworks and the compiler. XLA first applies target-independent optimization passes (common-subexpression elimination, fusion, partitioning for parallelism), then hardware-specific passes, and finally emits a binary optimized for TPU, GPU or CPU. It is this independence from the target that makes XLA portable: the same StableHLO graph can target several architectures.

Hold onto this point, it is the key to the whole piece. XLA was originally developed inside TensorFlow; it is now the heart of the open-source OpenXLA project, and it supports three framework frontends: TensorFlow, JAX and PyTorch. The (Tensor Processing Unit), for its part, only knows how to speak to XLA. So anything that wants to run on a TPU, JAX code as much as PyTorch code, ends up, one way or another, going through XLA. The question “how do you run PyTorch on a TPU” is therefore, at bottom, the question “how do you cleanly bring a PyTorch graph to XLA”. The rest of the piece describes the competing paths to get there.

JAX: the framework the TPU was designed for

is Google’s ML framework. For the non-specialist reader, it is “NumPy on steroids”: the same array API, augmented with automatic differentiation and compilation for GPU/TPU. For the engineer, it is a functional-programming library whose strength is the composability of four transformations: grad computes a function’s gradient (the building block of all learning); jit compiles the function through XLA; vmap automatically vectorizes a function written for a single vector so it operates on a batch, with no Python loop; pmap parallelizes it across several accelerators under an model (Single Program, Multiple Data).

The internal mechanism is worth unfolding, because it explains why JAX and the TPU get along so well. JAX does not “see” your values: it traces them. When you apply jit, JAX runs your function with symbolic values (tracers) to reconstruct the sequence of operations as a jaxpr, its functional intermediate representation, which it then lowers to StableHLO for XLA. The Python code describes a computation rather than executing it step by step. It is a deliberately lazy model, and it is exactly the kind of whole graph XLA knows how to optimize.

Why have TPUs historically been optimized for JAX? Because the two were tuned together. Reuters’ December 2025 reporting says it plainly: for years, TPUs were tuned first for JAX, Google’s favored internal framework, which created a “bottleneck” (the term is in quotes in the article) for customers already built on PyTorch. The production JAX stack rests on Pathways, the orchestration layer that lets a single JAX client drive computations across thousands of chips, beyond the boundaries of one TPU pod. It is the system that trains Gemini internally, and labs like Anthropic, xAI and Apple use JAX as one of their foundation-model building tools.

The problem, from Google’s vantage point, is the other side of the scale. PyTorch dominates research and, increasingly, production. According to the Linux Foundation’s Shaping the Future of Generative AI report, relayed by the PyTorch Foundation, PyTorch leads model training with 63% adoption; the same foundation puts the share of AI research implementations that rest on it at over 70%. It is this imbalance that Google is trying to neutralize: silicon cut for JAX, facing a code world that writes PyTorch.

PyTorch/XLA: the bridge that already exists today

Before TorchTPU, running PyTorch on a TPU went (and still goes) through PyTorch/XLA, the torch_xla package. It is a public product, installable right now (pip install torch==2.8.0 'torch_xla[tpu]==2.8.0'), unlike TorchTPU. Understanding its execution model is understanding the step TorchTPU wants to remove.

PyTorch/XLA rests on lazy tensors. When you write z = x + y on an XLA tensor, nothing runs: the operation is recorded into a symbolic graph. The actual computation happens only at a synchronization point, typically the call to xm.mark_step(), which “cuts” the accumulated graph, compiles it and runs it on the TPU. HuggingFace’s documentation captures the contrast well:

import torch
import torch_xla.core.xla_model as xm

device = xm.xla_device()              # an XLA device, not CUDA
x = torch.randn(2048, 2048, device=device)

for step in range(100):
    y = x @ x                         # nothing runs: the op is recorded
    x = torch.tanh(y)                 # the graph grows, still nothing on the TPU
    xm.mark_step()                    # here: cut the graph, compile, execute

# A print(x) BEFORE mark_step would force a device -> host materialization
# and break tracing: the model's ergonomic Achilles' heel.

It is powerful (letting XLA see a whole graph lets it fuse and optimize), but it is also the Achilles’ heel. Any operation that demands a concrete value (a print, a log, a checkpoint) forces a materialization and a device→host transfer that breaks tracing. And if the shapes change at every iteration (dynamic shapes), XLA recompiles a new graph each time: a performance sinkhole, avoided by padding to fixed shapes. The historical frustration shows through in a researcher’s comment on Hacker News about training research models on the old bridge: “a mess of undocumented behavior and bugs (silently hanging after 8 hours of training!)”.

Two version details, because they matter in production. Since the 2.7 release (official post dated May 2025, the repository’s C++11 notice carrying the date of March 18, 2025), C++11 ABI builds became the default; Google no longer ships pre-C++11 ABI wheels, and these new wheels improve tracing performance: on a Mixtral 8x7B on v5p-256, the repository documents 39% MFU versus 33% before. Support spans Python 3.9 to 3.13, with 2.9 nightlies available.

The point that matters most for what follows is written in black and white in the official pytorch/xla repository: “Once TorchTPU is public it will replace PyTorch/XLA.” PyTorch/XLA is not a competitor to TorchTPU; it is its predecessor, destined to be replaced.

TorchTPU: making the TPU “behave like PyTorch”

Here is the heart of the piece. TorchTPU is Google’s new engineering stack for running PyTorch natively on a TPU. Its history is short: an internal project first revealed by Reuters on December 17, 2025 (the codename had leaked), then detailed officially on April 7, 2026 on the Google Developers Blog. The guiding principle is that a developer should be able to take an existing PyTorch script, switch its device to tpu, and launch the training loop without touching a line of business logic.

The TPU should be an obvious choice for any PyTorch user to target. […] Getting access through PyTorch has always been difficult. We are changing that this year.

Lee Howes, Engineering Lead, TorchTPU, Google (remarks reported by The Stack)

Eager First: the philosophical break

Where PyTorch/XLA imposed the lazy model, TorchTPU establishes an “Eager First” philosophy. Technically, it builds on PyTorch’s PrivateUse1 interface, the extension point meant for wiring in a new device type. The integration goes deep enough that you handle real torch.Tensor objects that happen to live on a TPU. The team’s phrasing: “No subclasses, no wrappers; just ordinary, familiar PyTorch Tensors on a TPU.” No subclass, no wrapper: an ordinary PyTorch tensor that responds like one on CUDA.

import torch

# TorchTPU: PrivateUse1 interface, real torch.Tensor living on the TPU
x = torch.randn(2048, 2048, device="tpu")
y = x @ x                 # eager execution, like on CUDA
print(y.sum())            # the value is available, no mark_step

# Custom kernel: decorate a JAX function, TorchTPU wires it via Pallas
@torch_tpu.pallas.custom_jax_kernel
def fused_softmax(x):
    ...

Three eager modes cover the development cycle. Debug Eager runs one operation at a time, with CPU synchronization after each step. Slow, but precious for hunting shape mismatches, NaNs, out-of-memory crashes. Strict Eager dispatches op by op but asynchronously, to reproduce the default PyTorch experience. Fused Eager is the breakthrough: by automatic reflection over the operation stream, TorchTPU fuses steps on the fly into denser blocks before handing them to the TPU. The claimed gain is “a 50% to 100+% performance increase over Strict Eager, with no setup required”, no user configuration. The three modes share a Compilation Cache, single-host or persistent multi-host.

Static compilation: Dynamo → StableHLO → XLA

For peak performance, TorchTPU integrates natively with torch.compile, and the path it takes is a deliberate architectural choice. FX graph capture through Torch Dynamo, then, instead of routing to Torch Inductor (PyTorch’s default backend), use of XLA as the backend compiler, through a translation layer that maps PyTorch operators directly to StableHLO. Google’s justification: XLA is “rigorously battle-tested for TPU topologies” and knows how to natively optimize the overlap between dense compute and collective communications across the (Inter-Chip Interconnect). In other words, Google is not reinventing a TPU backend for PyTorch: it is rewiring PyTorch onto the TPU backend matured over years for JAX.

The MPMD problem PyTorch/XLA could not solve

The real architectural progress hides here, and it is less visible than the eager numbers. PyTorch/XLA supported only pure SPMD: all ranks run exactly the same code. But real PyTorch code diverges slightly from one rank to another: rank 0 typically does a bit of extra logging. This divergence, MPMD (Multiple Program, Multiple Data), broke TPU optimization, which is cut for SPMD. TorchTPU is architected to support these divergent executions by isolating the communication primitives where necessary, “at minimal cost”, while preserving the global view XLA needs. The concrete consequence: PyTorch’s DDP, FSDPv2 and DTensor distributed APIs work “out of the box”.

Portability does not erase the hardware, and Google gives the classic trap as an example. Many models hard-code an attention head dimension at 64, whereas current TPUs hit their peak matrix-multiply ( ) efficiency at 128 or 256. TorchTPU recommends a staged flow: first get a correct execution, then refactor suboptimal architectures or inject custom kernels. The fact that your code runs does not mean it runs well: hardware awareness stays your job.

vLLM: the serving layer

Everything above concerns training and general execution. vLLM occupies a distinct layer: production LLM inference serving. Born in 2023 at UC Berkeley’s Sky Computing Lab, it is the reference open-source inference engine, and its coupling with the TPU stack became, in late 2025, a signal about the real direction Google is taking.

Its central innovation, , attacks the memory waste of serving. To generate text, a transformer re-reads at every token the (the stored key and value tensors for the whole sequence); naive serving pre-reserves a contiguous block sized for the maximum length, and wastes most of the reserved VRAM. PagedAttention applies the idea of an operating system’s virtual memory: the KV cache is split into blocks (16 tokens per block by default, configurable), addressed through a table and stored non-contiguously. Fragmentation collapses. The broader story, offloading and routing included, is in our piece on the KV cache as the central object of serving.

Second pillar: . Instead of waiting for the whole batch to finish, vLLM works at the granularity of the iteration: as soon as a request completes, its KV blocks are freed and the next request in the queue is inserted at the following decode step. The accelerator stays saturated. The concept comes from Orca (OSDI 2022); the most-cited figure (up to 23× throughput) is Anyscale’s measurement against HuggingFace’s naive static batching, not a general gain, and it has to be presented as such.

This is where it connects to the thesis of this piece. vLLM is multi-backend: NVIDIA, AMD, Intel (CPU, GPU, Gaudi), IBM Power, AWS Trainium/Inferentia via plugin, and TPU. In October 2025, vLLM TPU was rebuilt around tpu-inference, a hardware plugin that unifies JAX and PyTorch under a single JAX→XLA lowering path. A model defined in PyTorch runs on a TPU with no code change, lowered through Torchax (a PyTorch→JAX bridge, now in its own repository google/torchax), while JAX is natively supported. The vLLM blog states it unambiguously:

The documented gains are real: on the previous path (PyTorch/XLA), the team had obtained +3.6× throughput on Llama 3.1-8B on v6e-1 and +2.1× on Llama 3.1-70B on v6e-8; the tpu-inference rebuild reaches “nearly 5x” the February 2025 prototype. A limit documented by Google Cloud frames the usage: on GKE, vLLM on TPU is single-host; multi-host is not supported, which for 400B+ models points to JetStream, Google’s multi-host TPU inference engine.

Note the detail that resonates with the rest of the piece. Even to serve PyTorch, the real path on a TPU runs through JAX→XLA via Torchax. On the serving side, the destination is not “PyTorch on TPU”: it is always XLA, reached through JAX.

How it all fits together: the mental model

Here is the edifice, top to bottom. This is the diagram to keep in mind.

Frameworks PyTorch · JAX what you write

Compiler XLA / StableHLO the pivot

Silicon TPU · ICI (torus) where it runs

Figure: The stack reads top to bottom: two competing frameworks at the top, a single compiler that reconciles them in the middle, a silicon that speaks only to that compiler at the bottom.

Layer 1: the frameworks, what the developer writes. Two worlds: PyTorch (imperative, eager, dominant) and JAX (functional, lazy, co-designed with the TPU). Competitors at the API level, they converge on the same compiler.

Layer 2: the compiler, the pivot. XLA, fed with StableHLO. JAX gets there through jit/jaxpr. PyTorch gets there through three historically distinct paths: the old PyTorch/XLA bridge (lazy tensors, mark_step); the new TorchTPU (eager-first, torch.compile → Dynamo → StableHLO → XLA, MPMD); for serving, Torchax (PyTorch → JAX → XLA) inside vLLM.

Layer 3: the silicon, where it runs. The TPU, which is not an isolated chip but a network: chips linked by the ICI in a 2D/3D torus topology, each combining (dense matmul) and SparseCores (embeddings, gather/scatter, collectives). XLA can also target GPU and CPU: that is where portability comes from.

Around them, the satellite tools, by tier of need. TorchTitan: the native PyTorch platform for large-scale training (composable parallelism through DTensor and DeviceMesh), a TorchTPU integration target for 2026. Pathways: the JAX-side orchestrator that extends an XLA computation beyond a pod. Helion: a “PyTorch with tiles” Python DSL that automates kernel autotuning. Pallas / Mosaic: the JAX custom-kernel language, which lowers through Mosaic onto the TPU, the way to extract the last drop of performance where XLA cannot yet optimize. MaxText / JetStream: the reference JAX implementation for LLM training, and the multi-host TPU inference engine.

The line to remember: PyTorch and JAX fight over the developer; XLA reconciles them on the silicon; vLLM serves the result. TorchTPU is the project that turns “PyTorch can technically run on a TPU” into “PyTorch runs on a TPU like it does at home”.

NVIDIA’s real moat is not the silicon

NVIDIA does not dominate because its chips work some inaccessible magic. They are excellent, full stop. It dominates because the default path runs through CUDA, and that path holds a decade’s head start: tooling, optimized kernels, network effect. Over 6.5 million developers use NVIDIA software (figure released in August 2025), more than 6 million of them on CUDA. It is a software moat, and it is what TorchTPU targets, not the peak FLOPS.

The numbers behind the moat are eloquent. In fiscal 2026 (closed at the end of January 2026), NVIDIA posted roughly $194 billion in datacenter revenue, up 68% year over year, about 90% of its total revenue (~$216 billion), with a non-GAAP gross margin on the order of 71% for the year (75% in Q4 alone). Networking revenue alone (NVLink, AI Ethernet) is near $30 billion. On market share, IDC puts NVIDIA around 81% of datacenter AI chips, its closest rival, AMD, holding only about 10%: an analyst figure, to be treated as such. As for revenue visibility, it was Jensen Huang, at GTC in October 2025, who quantified it (“the first technology company in history to have visibility into half a trillion dollars”) at roughly $500 billion of Blackwell + Rubin from early 2025 to the end of 2026, a figure since raised to some $1 trillion through the end of 2027.

It is this wall that the software friction sustains, and that is why the fight plays out at the compiler level, not the die. For the exact nature of this lock-in and how it compares to ROCm, see our piece on CUDA vs ROCm in 2026.

Google’s answer: portability as a weapon

Google’s idea is to turn the AI chip into a commodity by removing the switching cost. Three levers, all already engaged in 2026.

The first: make the TPU “PyTorch-friendly” with TorchTPU, so PyTorch shops no longer have to rewrite their stack in JAX. The second: ally with Meta, PyTorch’s creator and keeper. Reuters reports that Google works closely with Meta and is considering open-sourcing parts of TorchTPU to speed adoption. Meta, one of NVIDIA’s biggest customers (~$72 billion in 2025 capex), has a direct interest in lowering its costs and strengthening its negotiating leverage; the reporting indicates it explored renting TPU capacity as early as 2026 and deploying them in its own datacenters from 2027. The third: multi-stack portability (JAX, PyTorch/XLA, ONNX) that lets the same model train on NVIDIA, Google or AMD with minimal changes.

The business context is explosive. Sundar Pichai stated, on Alphabet’s Q3 2025 earnings, that Google Cloud had signed more deals worth over $1 billion in the first nine months of 2025 than in the previous two years combined. And above all Anthropic: the October 23, 2025 agreement grants access to “up to one million TPU chips… well over a gigawatt of capacity coming online in 2026”, valued at “tens of billions of dollars”. Thomas Kurian, CEO of Google Cloud, justified it this way:

Anthropic’s choice to significantly expand its usage of TPUs reflects the strong price-performance and efficiency its teams have seen with TPUs for several years.

Thomas Kurian, CEO, Google Cloud

The agreement was extended in April 2026: a Google investment in Anthropic of up to $40 billion (announced April 24, “up to”), and a next-generation capacity agreement through Broadcom: 3.5 GW per Broadcom’s regulatory filing, “up to 5 GW” in Google’s phrasing. A new development, finally: Google started selling TPUs for delivery to a restricted circle of customers in their own datacenters, where historically you could only rent them on Google Cloud.

Two chips for two jobs: the 8th generation

At Google Cloud Next, on April 22, 2026, Google unveiled an 8th-generation TPU split for the first time into two distinct chips. (Not to be confused: Ironwood, the 7th generation, had reached general availability back in late 2025; it is the 8t/8i split that is the April 2026 announcement.) The rationale: training and inference have diverged into two workload profiles, and a single silicon serves both poorly at once, the same phase-specialization logic seen on the NVIDIA side with Vera Rubin’s disaggregation.

Criterion	Ironwood (TPU v7)	TPU 8t (training)	TPU 8i (inference)
HBM per chip	192 GB	216 GB	288 GB
On-chip SRAM	n/a	128 MB	384 MB (×3)
Peak compute / chip	4,614 TFLOPS	12.6 PFLOPS FP4	10.1 PFLOPS FP4
HBM bandwidth	7.37 TB/s	6.53 TB/s	8.6 TB/s
Topology	3D torus	3D torus	Boardfly
Compute blocks	TensorCores + SparseCores	+ LLM Decoder Engine	2 TensorCores + 1 CAE
Scale	9,216-chip pod	9,600 superpod (Virgo fabric)	Boardfly ≤ 1,024 active
Perf/$ vs Ironwood	reference	up to 2.7× (training)	up to 80% (inference)
Native PyTorch	via PyTorch/XLA	TorchTPU (preview)	TorchTPU (preview)

Table 1: Ironwood (TPU v7) vs the split 8th generation. Google vendor specs (Cloud deep-dive, April 22, 2026). Ironwood's peak compute is its announced peak; the 8t/8i are figured in native FP4.

The TPU 8t, for training, bets on scale. Native that doubles MXU throughput, an Axion ARM host, and above all a fabric named Virgo that links over 134,000 chips with up to 47 Pb/s of non-blocking bisection bandwidth, for over 1,700 ExaFlops at near-linear scaling, enough to exceed one million chips in a training cluster, driven by JAX + Pathways. The claimed gain is up to 2.7× performance per dollar vs Ironwood (Google’s figure; several outlets cite 2.8× or even 2.85×, which are not the primary number). The superpod counts 9,600 chips, and Google’s consumer-facing post advances 121 ExaFlops and 2 PB of HBM shared per superpod.

The TPU 8i, for inference and reasoning, bets on latency. 288 GB of HBM and above all 384 MB of on-chip SRAM (×3 vs the previous generation) to house a larger KV cache as close as possible to the compute. The radical novelty is topological: dropping the 3D torus for Boardfly, which brings a 1,024-chip configuration’s network diameter down from 16 hops to 7 (−56%), for latency improved “up to 50%”. And a new fixed block, the CAE (Collectives Acceleration Engine): two TensorCores and one CAE per chip replace Ironwood’s four SparseCores, and cut on-chip collectives latency “by 5x”, decisive for autoregressive decoding and chain-of-thought. Claimed gain: up to 80% performance per dollar vs Ironwood in inference. The two chips share bare-metal access and up to 2× perf/watt, and both “run native PyTorch via TorchTPU in preview”. The 8th generation therefore arrives with TorchTPU as a front-line software argument.

A structural limit to keep in mind: the TPU was long rentable only on Google Cloud, never buyable standalone. NVIDIA GPUs run everywhere: multiple clouds, on-prem, workstations, edge. Google has started to correct this by selling TPUs to a few customers, but for a multi-cloud or sovereign strategy, the deployment-flexibility gap stays a decisive factor. A sign that Google is not trying to purely replace NVIDIA: it also offers NVIDIA Vera Rubin NVL72 instances on its own Virgo fabric.

How to choose for your situation

You are a PyTorch-first shop on GPU evaluating the TPU to cut the bill. Migrate nothing in preview. The public, stable path stays PyTorch/XLA (torch_xla[tpu]), not TorchTPU. Run an honest A/B test (the same PyTorch model on a GPU cluster and on a TPU slice) and compare sustained throughput, energy and bill. Watch two signals that change the game: the release of TorchTPU’s public GitHub repository with the flip to GA, and the validation of linear scaling at Pod size. Until these two milestones are cleared, TorchTPU is a credible promise, not a production foundation.

You do LLM serving and want the best cost per token. vLLM is the default option, multi-backend. On TPU, start from tpu-inference (single-host on GKE); for 400B+ multi-host models, switch to JetStream. Keep your model definitions in PyTorch: Torchax lowers them with no code change. The threshold that justifies seriously evaluating the TPU for inference: an MoE or reasoning workload at high concurrency with a low latency target, exactly the profile the 8i (CAE, big SRAM) was designed for.

You are starting a very large-scale research project on TPU. JAX + MaxText + Pathways stays the path of least resistance and the most mature: it is the stack that trains Gemini. Adopt PyTorch-on-TPU only if your code and talent capital is already massively PyTorch.

You have a multi-cloud, on-prem or sovereign constraint. The NVIDIA GPU stays the safe choice as long as TPUs are not widely buyable outside Google Cloud. Keep your pipelines portable (Keras 3, ONNX, framework-agnostic models) to switch between GPU and TPU as prices and availability evolve.

Conclusion

TorchTPU will not bring down the CUDA wall on its own. What it changes is the height of that wall: it turns “PyTorch can technically run on a TPU” into “PyTorch runs on a TPU like it does at home”. The day an infrastructure lead can say “we move significant workloads without heroic engineering”, the negotiating leverage flips. That is precisely what Google is trying to make true.

What remains is the question that decides everything, and that no spec will settle before the first GA deployments. Will TorchTPU’s maturity be enough to migrate the PyTorch-first shops? Or will JAX/Pathways stay the path of least resistance for whoever wants the maximum out of the TPU? One clue leans for TorchTPU: teams like Liquid AI moved from JAX to PyTorch in 2023 for ecosystem reasons, not code, proof that framework lock-in is more surmountable than people think. But one clue leans against: even vLLM TPU, to serve PyTorch, lowers it to JAX→XLA via Torchax. At least on the serving side, the real destination stays XLA reached through JAX. The real question for 2027 is therefore no longer “can the TPU run PyTorch”: it is how many developers will cross the bridge once it is stable, and which of them will find they were, without knowing it, already on the JAX side.

Sources and method

This piece draws on verification work closed on June 9, 2026. Every figure in it is, unless noted otherwise, a verified fact from a primary source.

Software primary sources:

OpenXLA: architecture and tf2xla. Definition of XLA, fusion (“Fusion is XLA’s single most important optimization”, verbatim), the reduce_sum(x + y*z) example, StableHLO as the portability layer, TF/JAX/PyTorch frontends.
JAX: key concepts. grad/jit/vmap/pmap, tracing → jaxpr → StableHLO.
PyTorch/XLA: pytorch/xla repository (install command, C++11 ABI, “Once TorchTPU is public it will replace PyTorch/XLA”, verbatim) and 2.7 release post. The lazy/mark_step model and the “CPU and CUDA tensors… are lazy” quote come from the HuggingFace PyTorch/XLA blog. The “mess of undocumented behavior and bugs” comment is from user in-silico on Hacker News.
TorchTPU: Google Developers Blog, April 7, 2026. “it should feel like PyTorch”, PrivateUse1, three eager modes, “50% to 100+%”, Dynamo→StableHLO→XLA, MPMD, roadmap, preview. The long Lee Howes quote (“The TPU should be an obvious choice…”) is reported by The Stack, not by the official post, which contains no direct quotes.
vLLM: tpu-inference blog, October 16, 2025 (verbatim fallback quote, +3.6×/+2.1×, “nearly 5x”); PagedAttention (design doc) and SOSP 2023 paper; Torchax on google/torchax; multi-host/JetStream on Google Cloud GKE.

Hardware and business primary sources:

TPU 8t/8i: Google Cloud technical deep-dive (April 22, 2026) and the associated blog.google post: all the Table 1 specs, Virgo, Boardfly, CAE, and the “up to” figures (2.7× training, 80% inference, 5× collectives, 50% latency, 56% diameter). Ironwood: Google blog (April 2025), GA reached in late 2025.
NVIDIA: FY2026 results (8-K / NVIDIA Newsroom) for datacenter revenue of ~$194 billion ($193.7 billion at the filing), +68%, ~90% of total revenue; the “half a trillion” visibility is Jensen Huang’s at GTC (October 2025). The ~81% market share (IDC) is an analyst estimate.
Deals: Anthropic/Google announcement (October 23, 2025, PR Newswire) (verbatim Kurian quote, “up to one million TPU chips”); the April 2026 extension (Google investment “up to $40B”, 3.5 GW capacity per the Broadcom filing / “up to 5 GW” per Google); Reuters (December 2025) for the Meta collaboration and the contemplated open-sourcing of TorchTPU. Liquid AI (JAX→PyTorch 2023): the founder’s public testimony.

Method notes and points to handle with care:

Vendor “up to” figures. All 8t/8i gains (perf/$, latency, perf/watt) are Google claims phrased as “up to”, not verified by an independent benchmark. Same for Fused Eager’s +50 to 100%.
Consistency notes. NVIDIA’s 75% gross margin is the Q4 figure, not the full year (~71% non-GAAP over the year). vLLM’s “23×” is a maximum against naive static batching (Anyscale measurement), not a general gain; the concept itself comes from Orca.
Secondary figures, outside Google’s primary source: the “Sunfish/Zebrafish” codenames, the Broadcom (training)/MediaTek (inference) split, the TSMC 2 nm node and a “2027” GA timeline do not appear in the official deep-dive (Google announces a GA “later this year”, i.e. 2026). The official post itself carries an internal inconsistency on the Boardfly chip count (“up to 1,152” vs “up to 1,024 active chips”).
Currencies. Amounts in this article are in US dollars, as reported by the cited sources (SEC filings, analyst notes, press releases). The French edition converts to euros at an indicative 1 USD = 0.92 EUR. Direct quotes (“half a trillion dollars”, “tens of billions of dollars”) are reproduced in their original currency.
Preview API. The TorchTPU code snippets (device="tpu", @torch_tpu.pallas.custom_jax_kernel) are illustrative: the API is in preview and may change before GA.
Header image. Google TPU v3 board photographed by Zinskauf, Wikimedia Commons, under CC BY-SA 4.0 license, cropped for the header (illustrative image of an earlier TPU generation).

A French version of this article, published on June 9, 2026, is the original.