AI Agents Fall Below 12% on Practical Benchmark

SemiAnalysis, a research firm in the semiconductor and AI sectors, has released a new benchmark called "GDPx" to measure the practical capabilities of AI agents. The benchmark evaluates AI across a broad range of domains simulating real business scenarios, from chip design to quantitative trading, with the highest-scoring model to date unable to exceed 12%.

An AI agent is an AI system that autonomously performs tasks upon receiving instructions. Recently, various companies have been launching products claiming to "handle complex tasks on behalf of humans," and industry-wide expectations for agents have been rising. However, evaluations often rely on simple metrics such as text generation accuracy or knowledge question-answering, and there have been long-standing concerns that proper benchmarks for measuring real-world utility are lacking. GDPx is positioned as a benchmark designed to address these concerns.

Claude Opus 5.5 from Anthropic topped this evaluation. Google's Gemini 4 Carbon followed closely behind. However, both models' scores fell short of 12%, with only a marginal difference between them. According to results published by SemiAnalysis, the major models tested are similarly clustered at low performance levels.

Nevertheless, there is also critical perspective on the benchmark itself. Questions have been raised about the small number of tasks assigned to each domain and inconsistencies in test conditions. With limited sample sizes, chance results could significantly influence scores, and it cannot be definitively said that the scores accurately reflect differences between models.

Yet the implications of these results are not insignificant. The findings explicitly demonstrate that current AI agents may possess only limited capabilities for complex tasks resembling real-world work. In highly specialized domains such as chip design and financial analysis, accuracy and multi-step reasoning are essential. If current models fall short in these areas, this could serve as material for reconsidering assumptions underlying the outlook that "AI will replace human work."

Going forward, a key question is whether GDPx will establish itself as an industry standard. If the benchmark's design is refined and task numbers and evaluation criteria are standardized, it could become a more reliable measure for assessing AI agent capabilities. How companies respond to this evaluation metric and how scores change with next-generation models will be important points to watch.