ScRPO: From Errors to Insights

The introduction of the Self-correction Relative Policy Optimization (ScRPO) framework marks a significant advancement in the field of reinforcement learning, particularly for enhancing large language models' capabilities in tackling challenging mathematical problems. ScRPO operates through a two-stage process: the trial-and-error learning stage, where the model is trained using GRPO and collects incorrect answers in an error pool, followed by the self-correction learning stage, which encourages the model to reflect on its mistakes. Extensive experiments conducted on multiple math reasoning benchmarks, including AIME, AMC, Olympiad, MATH-500, and GSM8k, demonstrate that ScRPO consistently outperforms several post-training methods. This promising paradigm not only highlights the potential for language models to self-improve on difficult tasks but also paves the way for the development of more reliable and capable AI systems that can operate effectively with limited external feedback.