Kwai AI Develops SRPO to Significantly Improve Reinforcement Learning Efficiency
China's Kwai AI research team has unveiled SRPO, a reinforcement learning framework for large language models. By combining two-stage reinforcement learning with historical data reuse, SRPO reduces learning steps by approximately 90% compared to the existing method GRPO, while achieving performance equivalent to DeepSeek-R1 in mathematics and code generation.
Kwai, a Chinese company operating a video platform, has unveiled SRPO, a new learning framework for reinforcement learning in large language models (LLMs). This framework reduces the number of learning steps required compared to conventional methods by approximately 90%, while achieving performance equivalent to DeepSeek-R1 on mathematics and code generation benchmarks.
The background lies in the limitations of GRPO, a reinforcement learning method widely used in post-training of LLMs in recent years. GRPO is a technique that enhances reasoning capabilities by providing correct and incorrect feedback to models, and is also adopted in high-performance models such as DeepSeek-R1. However, GRPO tends to require many learning steps, and challenges in computational cost and time have been pointed out. SRPO is positioned as a method designed to overcome this inefficiency.
The core of SRPO lies in the combination of 'two-stage reinforcement learning' and 'historical data reuse (history resampling)'. In typical reinforcement learning, only the latest outputs generated by the model are used for learning, but SRPO reuses data from past learning histories, extracting more learning benefits from the same computational resources. Through this design, learning steps can be reduced to approximately one-tenth compared to GRPO.
In terms of performance, it is reported that results comparable to DeepSeek-R1 have been achieved in two fields: mathematical problem solving and program code generation. DeepSeek-R1 is widely known as a reinforcement learning-based reasoning model, and being able to reproduce its performance with significantly fewer steps represents a noteworthy achievement from the perspective of learning efficiency.
The significance of this research extends beyond mere speed improvements. Reinforcement learning for LLMs requires enormous GPU resources and time, which has been one factor limiting high-performance model development to large corporations and well-funded research institutions. If learning efficiency improves significantly, the possibility of achieving high performance with lower costs expands, and the foundation for research and development can broaden.
Furthermore, the fact that Chinese technology companies such as Kwai continue to publish independent learning methods demonstrates that AI research competition is open to various players, not just major American corporations. The detailed reproducibility of SRPO and its generalizability to other tasks will be important benchmarks for evaluation as the method's details are disclosed and validated in the future.
A key point to watch going forward is how widely SRPO can be reproduced and applied by other researchers. The efficiency improvement of reinforcement learning is a technical theme that can impact the overall cost structure of LLM development, and its practical value is expected to become clearer as the method details are disclosed and verified.
This article is an original work independently written and edited by the AI issue editorial team based on factual reporting. © AI issue. Unauthorized reproduction, redistribution, or use for AI training is prohibited.