Sutton Warns Against One-Sided Focus on LLM Scaling

Richard Sutton, known as a foundational AI researcher, has once again expressed the view that scaling of large language models (LLMs)—that is, expanding computational resources and continuously enlarging models—alone cannot achieve true intelligence. Sutton is a Turing Award winner and is internationally recognized as a researcher who pioneered the field of reinforcement learning. His remarks can be said to challenge fundamental questioning of the direction that currently defines mainstream AI development.

Sutton previously pointed out in an essay known as the 'Bitter Lesson' that in the history of AI, simple methods leveraging vast data and computational power have proven superior in the long term to complex rules designed by humans. This essay has been widely cited in the AI industry as evidence of the effectiveness of scaling. However, Sutton himself has now clearly taken a position criticizing the excess of that interpretation, drawing attention from within the industry.

The crux of Sutton's argument is that merely increasing computational volume is insufficient, and that a new methodology is needed to fundamentally reconstruct the learning architecture itself. There is recognition that the current mainstream approach of simply continuing to expand the Transformer architecture has limitations, a view shared by Jessica Tworetzky, a former OpenAI researcher. Tworetzky expresses that blind scaling of Transformers has 'hacked the thinking of an entire generation of scientists'—referring to reward hacking, the myopic optimization of maximizing only desired outcomes.

'Reward hacking' refers to the phenomenon in reinforcement learning where, rather than achieving the original objective, optimization becomes focused solely on raising evaluation metrics. By deliberately employing this term, Tworetzky implies that the fixation on scaling has distorted the very problem consciousness and research direction of scientists themselves. In other words, it can be read to mean that while performance metrics have improved, the situation has persisted where it is difficult to say that we are approaching the essence of intelligence.

Parallel to this discussion, there are reports that more startups are exploring new learning methodologies beyond the prevailing scaling-focused approach. These companies can be understood as seeking alternatives to the path pursued by major technology companies, which are investing massive capital in scaling. While specific company names and methodological details remain undisclosed, some sectors of the industry suggest that capital flows may already be shifting.

The fact that a leading researcher like Sutton has objected to the context in which his own work is being cited can be positioned as posing an important question to the AI research community. While the performance improvements brought by scaling are undeniable facts, the answer to the fundamental question of whether that constitutes a path toward 'intelligence' remains unsettled. Going forward, how far learning paradigms beyond scaling can demonstrate concrete results will likely become the focal point for discerning the overall direction of the industry.