AI Coding Agents Hit Ceiling on Over 70% of Real-World Tasks

How applicable is AI in actual coding environments? The benchmark that directly addresses this question is 「Co-Design Bench」, released by China's Alibaba. Even the model with the highest performance on this benchmark achieved only a 26.4% complete task resolution rate. In other words, the remaining approximately three-quarters of tasks required human correction.

Various benchmarks have been used to evaluate AI coding tools. However, most follow a format where AI solves pre-prepared problems, which differs significantly from how these tools are actually used in development environments. As a result, researchers have frequently pointed out that even with high scores, it remains unclear whether such performance will translate to real-world business applications.

What makes Co-Design Bench distinctive is its approach to selecting test problems. It collects cases where actual users found that AI agents 「failed to work properly」 and uses these as test material. By using real-world failure cases directly as test content, the benchmark is designed to expose weaknesses that traditional evaluations failed to reveal. The model achieving the top score on this benchmark is reported to be 「GPT-4 Astra」, not 「GPT-4o」, with a complete resolution rate of 26.4%. Meanwhile, the same model scores over 99% on conventional, older-style benchmarks.

The gap between these numbers is highly significant. The disparity between over 99% and 26.4% suggests that conventional benchmarks may not adequately reflect actual difficulty. Put differently, high benchmark scores do not guarantee that a tool will work 「almost perfectly in real business applications」. In the current state, AI coding agents can be useful as auxiliary tools, but their capacity for autonomous task completion remains limited.

Expectations for AI agents are rising across the industry, with the coding field positioned at the forefront. Software development automation is an area where many companies are actively investing from a productivity improvement perspective, and the importance of accurately measuring agent capabilities is increasing. As 「real-world-aligned evaluation standards」 like Co-Design Bench become more widespread, developers and companies will find it easier to determine which tools best fit their needs.

Going forward, a key point to watch is how widely reality-based benchmarks become established as industry standards. While model performance competition has centered on benchmark scores, if evaluation axes focused on 「whether tools are actually useful」 become mainstream, they could influence development directions. The discussions surrounding the practical implementation of AI coding agents are entering a stage where the nature of benchmarks themselves are being reconsidered.