AI agents fell well short of reliably completing full computational-biology workflows in a preprint that tested 13 frontier models on 20 tasks. Across the benchmark, model scores ranged from 0.00 to 0.48. Because a task score was the proportion of requested artifacts that passed the benchmark's grading threshold, even the highest overall result represented fewer than half of the required outputs passing.
The authors describe a mixed picture. Agents could coherently execute many sequential computational-biology tasks, but they struggled with large datasets, long analysis chains and recovering from errors. The benchmark prescribed the analyses and methodological plans, so this was a test of carrying out specified work, not of deciding which biological questions or methods were worth pursuing.
A test of execution, not scientific judgment
BixBench3 was introduced to measure whether AI agents could take raw biological data through to scientific results in complete computational-biology workflows. Its 20 published-study tasks covered 17 assay types and nine scientific domains, with 138 graded artifacts.
Thirteen frontier models ran across those 20 tasks, producing 260 completed runs and 1,794 graded artifact evaluations. In the scoring system, an artifact passed when its composite score reached at least 0.8, and the task score was the share of artifacts that passed. That setup distinguishes a workflow that gets some outputs right from one that reproduces the requested analysis more completely.
Each agent ran in a controlled environment with 32 virtual CPUs, 128 GB of memory and a 500 GB disk, without a GPU. The wall-clock limit was 24 hours, and there was no fixed token limit.
Where the test became difficult
The clearest weakness appeared when the input data became very large. Mean task scores were 0.37 for tasks with less than 50 GB of raw input, 0.34 for those between 50 and 100 GB, and 0.10 for tasks above 100 GB. This is a descriptive comparison between task groups, so it does not establish that file size alone caused the lower score.
Analysis depth showed a more uneven pattern. Mean pass scores, based on whether individual artifacts passed or failed, were 0.30 for depth-one artifacts, 0.44 for depth-two artifacts and 0.24 for artifacts at depth three or greater. The deepest group therefore had the lowest average pass score, while the middle group had the highest.
Failure patterns tracked performance closely. The number of failure tags was strongly negatively associated with mean task score, with a Spearman correlation of -0.92 and a reported p-value of 9.9 × 10−6. Spearman correlation is a rank-based measure of association, so this result describes how the patterns moved together rather than showing that one caused the other. Premature termination was about twice as common in the lowest-scoring group, while repetitive retry loops were about 2.2 times as common.
More compute did not guarantee better scores
The work also consumed substantial resources. On average, a task took 6.8 hours, used 102 million tokens and cost $43. The longest attempts ran for 24 hours, used 1.07 billion tokens and cost $525.
Yet the highest performance occurred at intermediate levels of token use, runtime and model turns, rather than at the maximum levels reported. That means the longest or most expensive attempt was not automatically the strongest one, although the supplied analysis does not quantify the relationship.
A useful benchmark with a narrow remit
The researchers also checked whether the automated scores tracked human assessment. Experts reviewed 25 artifacts, rating reproduction from 1 to 5. Those ratings were strongly associated with the continuous artifact scores, with a Spearman correlation of 0.76 and a reported p-value of 1.3 × 10−5. The result gives some support to the automated scoring approach in that reviewed sample.
The benchmark's limits matter. Because it prescribes analyses and methodological plans, it tests execution of specified pipelines rather than selection of research questions or methods. Programmatic grading can also penalize a scientifically valid alternative when its output differs from the published artifact, and the benchmark inherits possible weaknesses in the source papers.
Taken together, the benchmark shows a capable but brittle form of execution. Agents can carry out many linked computational-biology steps, according to the authors, but the results expose difficulties with data scale, long chains and error recovery. The study's strongest conclusion is therefore about performance on prescribed workflows, not autonomous scientific judgment.
The paper is a preprint dated August 27, 2026. It states that the BixBench3 code is available on GitHub and the dataset on Hugging Face, with future updates on Edison Advances. All authors were employed by Edison Scientific for the duration of the study, and the supplied declaration reports no funding statement.
Paper data and sources
Original title: BixBench3: Benchmarking AI agents on research-study-scale computational biology tasks
Authors: Zane Koch, Asmamaw T. Wassie, Javier Valdes-Aleman et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text