OpenAI Points Out Defects in Approximately 30% of AI Coding Evaluation Metrics

OpenAI has verified the benchmark 'SWE-Bench Pro,' which is widely used to measure AI models' programming capabilities, and confirmed that approximately 30% of all tasks contain defects. In response, the company is withdrawing the support it has previously shown for SWE-Bench Pro.

SWE-Bench Pro is designed to evaluate a model's practical coding capabilities by having AI solve problems that mimic challenges arising in real-world software development. Such 'benchmarks' have been widely used by researchers and companies as a common standard for comparing and selecting AI models. Particularly in the coding field, because accurate assessment of programming assistance tools directly depends on these evaluations, industry adoption has been progressing.

What OpenAI has clarified is the fact that approximately 30% of the tasks included in SWE-Bench Pro are 'broken,' meaning they are in a state of not functioning properly. If the benchmark itself has defects, the evaluation of models based on those results may lack accuracy. OpenAI has decided to withdraw its recommendation and endorsement of the benchmark based on these verification results.

The significance of this issue is not small. The performance evaluation of AI models directly influences decisions about which models to adopt and which products to trust. If the benchmark that serves as the foundation for evaluation has major defects, the very basis of claims such as 'this model excels at coding' becomes unstable. Given that the entire industry has been comparing models using the same standard, a situation where that standard's reliability is called into question could have far-reaching implications.

Concerns about AI evaluation metrics have long included the problem that 'models that overfit to benchmarks do not perform as expected in real-world applications.' This case differs in nature from that issue and presents a more fundamental challenge, namely that defects exist in the design and operation of the evaluation problem itself. The task of establishing reliable evaluation standards has become increasingly important as AI implementation advances.

Going forward, attention will focus on how the correction of SWE-Bench Pro or the development of alternative benchmarks proceeds. Additionally, whether efforts to re-examine whether similar problems are hidden in other benchmarks will expand is positioned as an important perspective that will influence the evaluation credibility of the entire industry.