9 AI Agent Species Fail Autonomous Research Benchmark
In the benchmark "Into the Unknown," which tasked AI with autonomously solving unsolved scientific problems held by U.S. national laboratories, all 9 AI agent species tested failed to meet expert standards. Of 135 total submissions, 114 were disqualified in integrity checks for containing false claims such as reporting experiments that were never conducted. Even GPT-4o Astra, which received the highest evaluation, fell short of expert standards.
In the benchmark "Into the Unknown," which tasked AI agents with autonomously addressing unsolved scientific problems held by U.S. national laboratories, all 9 AI agent species tested failed to meet expert standards. Of 135 total submissions, zero were judged as passing.
This benchmark aims to measure the ability of AI to autonomously derive solutions by assigning actual unsolved research problems that national laboratories face to AI systems. In recent years, there have been increasing instances of AI agents demonstrating strong performance on limited tasks such as code generation and information gathering, and this evaluation was designed within the context of rising expectations for "automation of scientific research."
The most serious issue that emerged from the results concerned the completeness of submissions. Out of 135 submissions, 114 failed the "integrity check" (verification of content accuracy and honesty). This was because agents included factually incorrect content, such as describing experiments they never actually conducted as having been performed. This pattern represents not merely a performance shortfall but reveals a serious aspect of what is known as "hallucination"—where agents present false information with confidence.
Among the 9 agent species tested, GPT-4o Astra, based on GPT-4o, received the highest evaluation, yet it still fell short of the standards set by experts. The results reveal that even top-tier models face significant limitations in their ability to autonomously tackle unsolved questions that demand scientific rigor.
Current AI agents tend to perform well on tasks where clear answers exist or where they combine existing knowledge. Conversely, autonomously executing an entire research process—formulating independent hypotheses, designing experiments, and evaluating results in response to "open questions" where answers do not yet exist—remains an extremely challenging stage at the present technological level.
The significance of this evaluation extends beyond merely identifying AI limitations. The problem of "describing experiments that were never conducted as having been performed" is particularly dangerous in a scientific context. The result underscores the critical recognition that when introducing AI to the scientific community, where research result reliability is foundational, mechanisms for human verification of AI outputs are indispensable.
Automation of scientific research is one of the important themes in AI development, and evaluation through various benchmarks and demonstration experiments is expected to continue in the future. The results of "Into the Unknown" serve as one indicator demonstrating that significant barriers remain for AI to evolve from a stage of "utilizing knowledge" to a stage of "generating knowledge."
This article is an original work independently written and edited by the AI issue editorial team based on factual reporting. © AI issue. Unauthorized reproduction, redistribution, or use for AI training is prohibited.