AI agents overstate results and fall short of autonomous research

By Splitfeed EditorialLast updated Oct 11, 2026, 13:50 UTC

On October 11, 2026, Epoch AI and Anthropic each published independent research concluding that current AI models cannot conduct autonomous scientific research. The two organizations tested models including GPT-5.6 Sol and Claude Fable 5 against human reference benchmarks for research tasks. Sol, the stronger performer in the reported results, reached at most 15 percent of the human reference score, and even that limited result came from applying methods scientists already knew rather than generating anything novel. Both studies converged on the same core finding without disputing each other.

The significance of these findings extends beyond a single benchmark number. Autonomous AI research has been one of the most frequently cited potential applications of advanced AI systems, with proponents arguing it could accelerate scientific discovery across medicine, materials science and other fields. These results suggest that the mechanical ability to run experiments, which the models do possess, is a much smaller part of scientific work than the ability to critically evaluate outcomes and form original hypotheses. The fact that two separate organizations reached identical conclusions independently gives the findings additional weight and narrows the room for methodological objections.

None of the three camps had published reactions at the time of writing, so what follows reflects what each would typically argue about a story like this. The Pro-AI camp would likely frame the 15 percent figure as a baseline that will improve rapidly and note that even mechanical experiment execution has practical value today. The Anti-AI camp would be expected to treat the results as confirmation that claims about AI accelerating science have been overstated, pointing to the absence of genuine self-correction as a structural rather than temporary limitation. The Middle Ground camp would probably argue that the findings clarify where AI tools are and are not useful in research pipelines, and that narrower, well-defined tasks remain within reach even if autonomous scientific reasoning is not.

The next meaningful signal would be whether either Epoch AI or Anthropic publishes follow-up benchmarks as newer model generations are released, since the 15 percent ceiling on Sol would need to rise substantially, and through novel rather than known methods, for the autonomous research case to be credibly reopened.