A tweet at 11:41 pm
On the evening of July 8, OpenAI’s official account posts a message few vendors would dare write: “We audited SWE-Bench Pro, one of the most used AI coding benchmarks, and found that it no longer reliably measures coding capability at the frontier. We estimate that 30% of its tasks are broken, and we retract our earlier recommendation to use it as a reference evaluation.” The next morning, the same company launched GPT-5.6, its new coding model.
You have to savor the situation in all its depth. SWE-Bench Pro is not an obscure benchmark: it is the one the trade press, comparison tables and discussion threads cite first to separate coding agents. It is also the one OpenAI had officially been recommending since February. And February, precisely: it was already the second time. Five months earlier, the same company had declared the predecessor, SWE-bench Verified, unusable, and advised migrating to Pro. In five months, the sector’s leading lab has therefore declared broken, one after the other, both reference instruments of its own discipline, including the one it had itself crowned.
The temptation is to read a scandal into it, rigged numbers, a maneuver. The reality is more interesting and more uncomfortable: it is a story of metrology. Code benchmarks did not become bad; they went obsolete under the progress of what they measured, and a second variable, long ignored, started to weigh more heavily than the model itself. This report takes the mechanism apart, piece by piece, because it concerns anyone who has ever chosen a model on the strength of a score table.
How a benchmark task breaks
Let us start with what SWE-Bench Pro measures, because the principle is sound. You give the model a real code repository and a real ticket (“this parser mishandles such a case”), it must produce a patch, and hidden tests decide whether the patch is good. No multiple choice, no artificial puzzle: engineering work, judged as in production. On paper, it is the best protocol we know.
The problem is industrial: these tasks are mass-produced, extracted programmatically from real code changes. But a real code change is not designed to become an exam question. OpenAI’s audit sorts the breakages into four families, and each deserves a moment.
The biggest: overly strict tests. The ticket asks to fix a behavior, the hidden tests additionally check implementation details that no one specified. A perfectly correct solution fails because it is not the historical solution. Then comes its mirror image, the under-specified statement: requirements do exist, but only in the tests, invisible from the ticket. The example flagged by the audit is telling: a task from the OpenLibrary project shows a string with a single space in the statement, and the hidden test requires two. No human engineer would pass that kind of test other than by luck. Third family, the exact reverse: tests with too little coverage, which let incomplete patches through. And finally the misleading statement, which actively steers toward the wrong approach.
Note what these four families have in common: none of them talks about the model. They are task-design defects, statements and tests that do not align. This is an important point, because the February diagnosis on Verified was different (we are coming to it): code benchmarks do not all die of the same disease.
The noise ceiling, or why an instrument goes obsolete
OpenAI’s wording deserves a word-by-word reading: SWE-Bench Pro “no longer measures” the frontier reliably. No longer. So it did measure something, for a time. What happened in between? Models improved.
The numbers tell the story better than any speech: on the public set of 731 tasks, the best models went from 23.3% success to 80.3% in eight months. But if roughly 30% of the tasks are broken, the maximum score actually reachable by a perfect agent tops out somewhere around 70%. As long as models struggle at 25 or 40%, this ceiling bothers no one: the gaps between models play out on the valid tasks, and the instrument ranks correctly. But as soon as scores approach the ceiling, everything flips. The points left to win are, for the most part, defective tasks; separating them amounts to measuring which model best guesses the whims of badly designed tests. Like a saturated sensor, the instrument crushes everything above into the same flat zone, and the gaps you read there are noise.
This is the central mechanism of this report, and it is general: a benchmark is not true or false in the absolute, it is usable within a range. The 2026 frontier has simply left the useful range of benchmarks designed in 2024. They were not wrong; they were surpassed, in the literal sense.
The audit itself is worth the detour, for its method and its irony. OpenAI first ran an analysis pipeline over the models’ attempts, their failure traces and the task metadata, with Codex-based investigator agents, able to open the real repositories and run the tests. This pipeline flagged 200 broken tasks (27.4%). In parallel, five experienced engineers ran a human annotation campaign, which identified 249 (34.1%), with 74% agreement between the two approaches, hence the median estimate of about 30%. The method is serious. The irony too: it was Codex agents that audited the benchmark tasked with evaluating Codex. The judged tool now certifies the measuring instrument; you could not better illustrate where the discipline stands.
Three generations of instruments, three deaths: the timeline
Let us pick up the thread from the start, because on one hand the exact scopes of each figure are precisely what the press mangles most, and on the other the July retraction is not the second episode of this story. It is the third.
| Instrument | Born | Declared broken | Main cause |
|---|---|---|---|
| SWE-bench (Princeton) | late 2023 | 2024, by OpenAI | unsolvable or under-specified tasks |
| SWE-bench Verified (OpenAI) | August 2024, 500 hand-sorted tasks | February 23, 2026 | test defects + contamination + saturation |
| SWE-Bench Pro (Scale AI) | September 2025, anti-contamination design | July 8, 2026 | ~30% broken tasks (design) |
At the origin, there is plain SWE-bench, created by Princeton researchers in late 2023: the founding idea, brilliant, of turning real GitHub tickets into exam questions. By 2024, OpenAI finds the benchmark riddled with unsolvable or under-specified tasks, and publishes the remedy: SWE-bench Verified, 500 tasks hand-sorted by professional developers paid to reread each statement. Keep this detail, it gives the rest its true depth: Verified was not one benchmark among others, it was already the corrected version, human-validated, co-built by OpenAI itself. It is that remedy which breaks in February 2026. And SWE-Bench Pro, designed at Scale AI precisely to resist the next ailment ( contamination The presence of a benchmark's data (problems, solutions) in a model's training corpus. The score then measures memorization, not capability: the model recognizes the test instead of solving it. Inevitable in the long run for any public benchmark built from public repositories. , with repositories under deterrent licenses and a private set), is the remedy’s remedy. It will have lasted less than a year. Three generations, three ever more careful designs, three ever faster obsolescences: this is not a series of accidents, it is a race between the instruments and what they measure, and the instruments are losing it.
On February 23, 2026, then, OpenAI publishes “Why we no longer evaluate SWE-bench Verified.” The audit does not cover the whole benchmark: it targets the 138 problems (out of 500) that its o3 model repeatedly failed over 64 attempts, the hard subset. Verdict: at least 59.4% of that subset has substantial defects, tests or description, including 35.5% of overly strict tests. Related to the entire benchmark, that comes to at least 16% broken tasks, not 59: the spectacular figure is true, but on a scope that must be cited with it. The symptom that had triggered the audit speaks for itself: model progress there was stalling, from 74.9 to 80.9% in six months, the characteristic flattening of an instrument hitting its limit.
Added to this was an ailment Pro does not have to the same degree: contamination. A paper accepted at NeurIPS, with a limpid title (“The SWE-Bench Illusion”), had shown that the best models recovered the file to fix in 76% of cases from the ticket text alone, without even seeing the repository, against 53% on tasks from repositories outside the benchmark. The models did not always solve the problem; sometimes, they remembered it.
On March 10, the METR evaluation institute adds the missing piece: even when the tests pass, is the work good? Four maintainers of major open-source projects reread 296 AI-generated patches declared “resolved” by the automatic grader. Their merge verdict is about 24 points below the benchmark scores: around 72% of “successful” tasks, about 48% of patches a maintainer would actually accept. Half of the green displayed would not have passed a human code review, for the usual reasons: missed functionality, regression elsewhere, insufficient quality.
And on July 8, then, Pro falls in turn, for task-design reasons this time. Three different diagnoses in five months (contamination, test/mergeable gap, defective tasks), three angles on the same finding: at the frontier, the displayed number and real capability have divorced.
The hidden variable worth a model generation
There is still a floor missing from the explanation, and it is the most counter-intuitive. Grant a benchmark with perfect tasks: would the score obtained finally measure “the model”? Not that either. Because a model never sits a benchmark alone: it sits it in a harness, that gear of tools, system prompt, execution loop and context management we dissected in the report on OpenAI’s comeback. And the harness effect on scores is not a second-order detail. It is a term of the same order of magnitude as the progress of a model generation.
The measurements pile up:
A dedicated research paper (Claw-SWE-Bench) quantifies gaps of up to 27.4 points of success between harnesses, on the same model and the same 350 tasks. Endor Labs measured GPT-5.5 at 61.5% in Codex’s native harness and 87.2% in Cursor’s, the same week: 25.7 points. An independent test swung Opus from 42% with minimal scaffolding to 78% with Claude Code’s full harness: 36 points, without touching the model. LangChain gained 13.7 points on Terminal Bench by changing only its harness, model strictly identical. The trade press settles on a range of 10 to 20 points of routine gap on SWE-bench at constant model, and OpenAI provides an involuntary demonstration in the GPT-5.6 announcement itself: Sol alone gets 88.8% on Terminal-Bench 2.1, and 91.9% when its “ultra” mode coordinates four agents. Same model, plus three points, all it took was changing the orchestration.
And even at a fixed model-harness pair, ordinary variance remains. An agent is not deterministic: from one run to the next, it explores differently, fails or passes borderline tasks. Serious protocols average several random seeds (OpenAI’s cyber evaluations use five per measurement); the press tables, meanwhile, often compare single runs. Two or three points of gap between models, on one run, with no confidence interval: that is not a ranking, it is a draw.
The consequence is brutal for how we talk about these systems: “model X’s score” does not exist. Only the score of a model-harness pair exists, in a given configuration, on a given date, averaged or not. When two tables give incompatible numbers for “the same model”, there is a good chance both are accurate.
That leaves the question of why the instruments go obsolete faster and faster, and the answer lies in an old law of measurement: when a metric becomes a target, it ceases to be a good metric. A public reference benchmark is not a neutral thermometer set beside the market; it is the target, explicit or not, of labs worth hundreds of billions that communicate in points gained. Each training cycle absorbs a little more of the benchmark, through direct contamination of the corpora or through simple selection pressure on what scores. The faster the industry optimizes, the shorter the useful life of a public yardstick. The original SWE-bench did not last a year before requiring its corrected version; Verified lasted eighteen months; Pro, less than a year. The rest is predictable.
What vendors publish, and what to make of it
This landscape casts a particular light on the July 9 communication. Far from skipping benchmarks, the GPT-5.6 announcement lines up the full range: the classic academic ones (GPQA, AIME, MMLU) as well as the agentic ones. It even publishes a figure on that very SWE-Bench Pro the company had just declared broken, and on which Sol actually stays behind Fable 5, 64.6% against 80%. Two anchors, however, call for caution: Artificial Analysis’s index, given as a third party, is an OpenAI pre-evaluation partner, a quality judge more than an independent one; and a footnote specifies that the latencies and costs shown are simulated offline, not measured in production.
The sequence also has to be reported in its exact order, because it feeds a reading you can neither assert nor ignore. On SWE-Bench Pro, Anthropic’s models dominated clearly: 69.2% for Opus 4.8 against 58.6% for GPT-5.5 in the spring. The benchmark where OpenAI was most visibly losing is the one it disavowed on July 8, the eve of its model’s launch. In its favor, the next day’s announcement did not hide the figure: the unfavorable score is there, disavowal included. If there is a maneuver, it is therefore subtler: publish the number and discredit the yardstick in the same gesture. There are the dated facts; the strategic reading remains a hypothesis, and honesty requires saying that the methodological critique, for its part, is solid, corroborated by independent work with nothing to sell (the NeurIPS paper on contamination, the METR note). Add the full irony: the retraction spares no one. Anthropic’s 69.2% rests on the same instrument declared broken; the scores brandished one way or the other now float on the same 30% of defective tasks. No one comes out taller from a warped yardstick.
Measuring yourself
What is left, then? OpenAI’s recommendation fits in one line and it is demanding: benchmarks built directly by experienced engineers, with human supervision integrated from the design stage, not generated in bulk then verified after the fact. Concretely, this looks like the GDPval-type evaluations the company now promotes: tasks written one by one by professionals in the field, with a rubric, judged by humans, at an accepted production cost. It is the admission that quality measurement does not industrialize: what made SWE-bench appealing (thousands of free tasks extracted from GitHub) is exactly what killed it. No successor is anointed for all that; the discipline is starting over from the drawing board.
In the meantime, two families of tools keep their value, precisely because they have internalized the problem. The independent trackers, first: Marginlab runs Codex daily on a contamination-resistant subset of tasks, fifty instances a day, with no custom harness, with a statistical test before declaring a drift; it is this protocol that detected an 8-point degradation of Claude Code in late May, invisible in static benchmarks. Scale SEAL, which we were already citing in the GLM-5.2 report, plays the same role of standardized yardstick. Their value comes not from miraculously perfect tasks, but from the protocol: constant conditions, repeated measurements, significance thresholds. Metrology, in the full sense.
The second family is yours. The teams that come out best have stopped reading other people’s tables and built themselves internal evaluations on their own code: real defects injected into their own pull requests, their languages, their conventions, judged on what a human reviewer would accept. On a smaller scale, the minimal protocol fits in one sentence: before switching model or tool, take five real tasks from your backlog, run them in A/B, measure review time and error rate, and decide on that. Five tasks from your repository are worth more than seven hundred broken tasks from someone else’s.
A method confession, to finish, because it illustrates the reflex to install. Our GLM-5.2 report, published three weeks before the retraction, cited SWE-Bench Pro scores. We had labeled them vendor figures, not replicated, with the independent aggregate index as the only level anchor; that is what lets them age with dignity today: the label survived the instrument. It is exactly the contract that coding-agent measurement now imposes on everyone, vendors, press and readers: each figure with its scope, its date, its harness and its judge. The frontier advances faster than its instruments; there is no reason for that to stop, and the benchmarks that replace SWE-Bench Pro will go obsolete in turn. Measurement is no longer a reading, it has become work again. Yours, preferably.
Sources and method
Editorial freeze date: July 11, 2026. Labels: verified fact (primary source), estimate (third-party analysis), hypothesis (assumed reading).
The July 8 retraction
- Fact: OpenAI, Separating signal from noise in coding evaluations (openai.com, July 8, 2026) and official @OpenAI post on X (July 8, 11:41 pm): 731 tasks, pipeline 200 (27.4%), human review 249 (34.1%), 74% agreement, ~30% estimate; four defect families; progress 23.3 → 80.3% in eight months.
The genealogy of the instruments
- Fact: SWE-bench: Jimenez et al., Princeton (arXiv 2310.06770, ICLR 2024). SWE-bench Verified: OpenAI, Introducing SWE-bench Verified (August 2024, 500 human-validated tasks). SWE-Bench Pro: Scale AI (September 2025, anti-contamination design).
- Fact: OpenAI, Why we no longer evaluate SWE-bench Verified (openai.com, February 23, 2026): 138/500 problems audited, ≥ 59.4% of that subset defective (35.5% overly strict tests), stagnation 74.9 → 80.9%.
- Fact: The SWE-Bench Illusion (arXiv 2506.12286, Purdue/Microsoft, NeurIPS 2025): 76% identification of the faulty file from the ticket alone, 53% outside the benchmark, verbatim similarity 35% against 18%.
- Fact: METR, Many SWE-bench-passing PRs would not be merged into main (metr.org, March 10, 2026): 296 PRs reread by 4 maintainers, verdict ~24 points below the automatic grader, “golden” baseline 68%.
The harness effect and variance
- Fact: Claw-SWE-Bench (arXiv 2606.12344): up to 27.4 points between harnesses at constant model. Endor Labs: GPT-5.5 at 61.5% (native Codex) against 87.2% (Cursor), same week. LangChain, Improving Deep Agents with harness engineering: +13.7 points at fixed model.
- Estimate: Opus from 42 to 78% depending on scaffolding (independent test published in April 2026); “10-20 points” range (trade press). The “5-15 points” range going around has no identifiable primary source: we do not repeat it.
- Fact: GPT-5.6 announcement (openai.com, July 9): SWE-Bench Pro Sol 64.6% against Fable 5 80%; Terminal-Bench 2.1 at 88.8% (Sol) against 91.9% (ultra mode, 4 agents); latencies and costs “simulated” (footnote 4); cyber evaluations at 5 seeds per measurement; Artificial Analysis pre-evaluation partner.
Scores and trackers
- Fact: SWE-Bench Pro: Opus 4.8 at 69.2%, GPT-5.5 at 58.6% (April GPT-5.5 post and trackers); Sol at 64.6% against 80% for Fable 5 (July 9 GPT-5.6 post, published alongside the instrument’s disavowal). Marginlab: daily protocol, N = 50, p < 0.05, 8-point Claude Code degradation detected in late May 2026. Scale SEAL: standardized yardstick (cf. GLM-5.2 report).
Method note. This article reports the July 8-9 sequence (retraction on the eve of launch) as dated facts; any strategic reading is flagged as a hypothesis. The scopes of the percentages (59.4% on Verified’s hard subset; ~30% on Pro’s public set) are cited with each figure, because their confusion is precisely the deformation mode this report documents. This article is the English adaptation of a French original, published on July 13, 2026.
