That distinction matters more than it sounds. As AI systems move from writing text to running actual analyses — pulling apart datasets, choosing statistical methods, deciding when a result is solid enough to act on — the interesting question stops being "how much does the model know" and becomes "how well does it reason under uncertainty." GeneBench-Pro is one of the first serious attempts to measure that directly, and its design reveals just how tricky it is to test judgment fairly.
A benchmark built to resist gaming
Earlier science benchmarks often ran into a quiet problem: when a test is built on real historical data, there may be several defensible ways to analyze it. If a benchmark author picked one cutoff or one method as "correct," a model choosing an equally reasonable alternative would simply be marked wrong — not because it reasoned poorly, but because it reasoned differently than the person writing the test. The opposite failure is just as damaging: if a problem doesn’t respond sharply enough to a wrong method, a model can botch the analysis and still stumble into a passing number.
OpenAI’s fix is to build every problem synthetically. Instead of collecting a real dataset and hoping its quirks are fair, researchers construct the entire "causal structure" — the true underlying biological relationships — and then generate messy, realistic-looking data from that known structure themselves. Because the ground truth is fully known in advance, grading can be deterministic: a given answer either matches the verified target or it doesn’t, with no rubric or human judgment call needed to decide who’s right. That also lets OpenAI stress-test each problem, deliberately checking that a plausible but wrong approach actually fails, rather than accidentally passing.
Synthetic data isn’t a trick unique to biology — it simply means data generated by a process designed to imitate the structure of the real world rather than harvested directly from it. That approach reduces one kind of ambiguity — is the target answer well-defined — but it doesn’t eliminate every source of bias or disagreement, and OpenAI’s own reviewers still had to check for realism and identifiability by hand, sending 82 of the 129 questions to outside experts including graduate students and industry scientists.
| Feature | Typical knowledge benchmark | Historical-data biology benchmark | GeneBench-Pro |
|---|---|---|---|
| What it mainly tests | Recall of facts or definitions | Executing a known workflow | Choosing the right approach under ambiguity |
| Data source | Curated or textbook examples | Real historical datasets | Synthetically generated, causally known |
| Grading method | Multiple choice or fixed answer | Rubric or author’s preferred path | Deterministic match to a verified target |
| Main failure risk | Too narrow to matter | Penalizes equally valid alternative paths | Requires ongoing checks that the "correct" path is truly identifiable |
| What a high score signals | Broad recall | Procedural competence | Consistent judgment across ambiguous steps |
Following one problem from mess to grade
Each GeneBench-Pro task follows roughly the same arc: an agent gets a short description of a scientific question, a data folder full of quirks, and a standard analysis toolkit (Python, statistical libraries, and genomics packages like PLINK). From there, it has to work the problem the way a researcher would — poke at the data, notice something’s wrong, adjust, and eventually commit to a number.
flowchart TD A[Messy dataset and short context] --> B[Exploratory data checks] B --> C[Choose analytical method] C --> D[Test and revise if wrong] D --> E[Final numeric answer] E --> F[Deterministic grading vs known truth]
That last step is what makes the benchmark’s design distinctive: because the "known truth" was defined before the data were even generated, there’s no room for the grader to prefer one defensible interpretation over another.
Where the models actually stumble
The headline number is that OpenAI’s strongest model, GPT-5.6 Sol, solved 28.7% of GeneBench-Pro problems at its highest reasoning setting, rising to 31.5% with an extended "Pro" mode. That’s a big jump from the original, easier GeneBench, where OpenAI’s best model at the time scored under 5% — but it also means roughly seven in ten problems still went unsolved. Other model families lagged further behind: reporting on the release notes Anthropic’s Claude Opus 4.8 reaching 16.0% and Google’s Gemini 3.1 Pro scoring 3.1% on the same set.
The more revealing detail is how models fail. External reviewers described what’s been called a "noticing-to-acting gap": models often correctly flag that something is wrong with the data — a mislabeled sample, a suspicious skew — but then fail to actually change their analysis in response. They notice the crack in the foundation and keep building on top of it anyway. That’s a strikingly human-sounding failure. It’s the same gap that often separates a first-year graduate student, who observes an anomaly but doesn’t know what to do about it, from an experienced scientist who reflexively adjusts the entire analytical plan.
Read the scoreboard carefully
It’s worth being honest about the limits of what this release shows. OpenAI built GeneBench-Pro, evaluated its own models against it, and used those same frontier models to help harden the problems during development — a dynamic the company itself flags as a possible source of bias. The full 129-question set also isn’t public; only ten sample problems are posted on Hugging Face, so outside researchers can’t yet audit the difficulty distribution themselves. A 50-question subset is slated for independent testing through Artificial Analysis, but that check hasn’t been published yet. Until it is, the current scores are best read as a vendor’s internal report card, not a settled, neutral ranking.
There’s also a cost argument layered on top of the accuracy numbers: reviewers estimated a typical problem takes a human expert 20 to 40 hours, while running the same problem through an AI agent costs only a few dollars. Even with a sub-third pass rate, that gap is large enough that partial automation — letting AI take the parts of an analysis it handles reliably and flagging the rest for a human — could still be economically meaningful. But cost savings and reliability are different questions, and a benchmark score, however designed, is not a guarantee of how a system behaves inside a real lab or hospital.
The real signal
GeneBench-Pro’s most useful contribution isn’t a single leaderboard number — it’s a demonstration that evaluating AI on messy, real-world-style data requires as much design discipline as building the AI itself. Sequencing costs have collapsed over the past two decades, and biobanks now hold vast troves of linked molecular and health data; the bottleneck genuinely does seem to be shifting toward analysis rather than data collection. But if a model can’t reliably notice a data problem and act on it, that shortfall matters more than any single accuracy percentage. The lesson for anyone watching AI move into scientific and data-heavy work isn’t that judgment has been solved — it’s that we’re finally building tests sharp enough to see how far it hasn’t been.
