Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning

A new study explores effective strategies for training large language models (LLMs) as agents through multi-turn reinforcement learning, identifying key design elements such as environment, reward, and policy. The research empirically tests frameworks like TextWorld, ALFWorld, and SWE-Gym to derive a systematic approach to training LLMs in complex tasks.

A-3PO, a new approach to asynchronous reinforcement learning (RL), has been introduced to enhance the training of large language models (LLMs) by reducing computational overhead. This method approximates the proximal policy through interpolation, eliminating the need for an extra forward pass, which traditionally slows down training. As a result, A-3PO achieves an 18% reduction in training time while maintaining performance levels comparable to existing algorithms.

A systematic comparison of three Reinforcement Learning algorithms—PPO, GRPO, and DAPO—has been conducted to enhance reasoning capabilities in large language models (LLMs). The study involved fine-tuning models on the Countdown Game and evaluating their performance on various reasoning benchmarks, revealing that RL-trained models generally outperform their base counterparts, albeit with varying degrees of improvement across benchmarks.