Comparative Analysis and Parametric Tuning of PPO, GRPO, and DAPO for LLM Reasoning Enhancement

arXiv — cs.LG•Tuesday, December 9, 2025 at 5:00:00 AM

PositiveArtificial Intelligence

A systematic comparison of three Reinforcement Learning algorithms—PPO, GRPO, and DAPO—has been conducted to enhance reasoning capabilities in large language models (LLMs). The study involved fine-tuning models on the Countdown Game and evaluating their performance on various reasoning benchmarks, revealing that RL-trained models generally outperform their base counterparts, albeit with varying degrees of improvement across benchmarks.
This development is significant as it provides practical insights into the training dynamics of LLMs, particularly highlighting how adjustments in group size can lead to more stable training and improved accuracy. The findings also indicate that disabling the Dynamic Sampling component in DAPO yields the best results, which could influence future model training strategies.
The exploration of different RL algorithms underscores ongoing challenges in optimizing LLM performance, particularly regarding stability and effectiveness. Issues such as the Lazy Likelihood Displacement in GRPO and the introduction of new frameworks like DVPO and GAPO reflect a broader trend in the field towards refining reinforcement learning methods to address specific shortcomings, ultimately aiming for more robust and capable AI systems.

— via World Pulse Now AI Editorial System

Read Original

Was this article worth reading? Share it

LucidQuery AI

Combines diffusion reasoning with autoregressive LLM for advanced AI analysis.

AI & DataView app details

Augmeta

AI peers for collaborative problem-solving and enhanced team productivity.

AI & DataView app details

Keywords AI

Monitor and optimize your AI models with comprehensive observability tools.

Business & ProductivityView app details

Continue Readings

arXiv — cs.CL2 days ago

A Systematic Evaluation of Preference Aggregation in Federated RLHF for Pluralistic Alignment of LLMs

PositiveArtificial Intelligence

A recent study has introduced a systematic evaluation framework for aligning large language models (LLMs) with diverse human preferences in federated learning environments. This framework assesses the trade-off between alignment quality and fairness using various aggregation strategies for human preferences, including a novel adaptive scheme that adjusts preference weights based on historical performance.

Read full article

via arXiv — cs.CL

arXiv — cs.LG3 days ago

A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning

NeutralArtificial Intelligence

A new study explores effective strategies for training large language models (LLMs) as agents through multi-turn reinforcement learning, identifying key design elements such as environment, reward, and policy. The research empirically tests frameworks like TextWorld, ALFWorld, and SWE-Gym to derive a systematic approach to training LLMs in complex tasks.

Read full article

via arXiv — cs.LG

arXiv — cs.LG3 days ago

Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning

PositiveArtificial Intelligence

A new paper proposes a novel approach to decoding-based regression by utilizing Reinforcement Learning (RL) to enhance numerical prediction accuracy. This method addresses the limitations of traditional token-level objectives, which often misalign with continuous numerical values, thereby improving the precision and generalization of predictions.

Read full article

via arXiv — cs.LG

arXiv — cs.LG3 days ago

A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation

PositiveArtificial Intelligence

A-3PO, a new approach to asynchronous reinforcement learning (RL), has been introduced to enhance the training of large language models (LLMs) by reducing computational overhead. This method approximates the proximal policy through interpolation, eliminating the need for an extra forward pass, which traditionally slows down training. As a result, A-3PO achieves an 18% reduction in training time while maintaining performance levels comparable to existing algorithms.

Read full article

via arXiv — cs.LG

arXiv — cs.LG3 days ago

Parent-Guided Semantic Reward Model (PGSRM): Embedding-Based Reward Functions for Reinforcement Learning of Transformer Language Models

PositiveArtificial Intelligence

The Parent-Guided Semantic Reward Model (PGSRM) has been introduced as a novel framework for reinforcement learning in transformer language models, utilizing cosine similarity between output embeddings of parent and child models to generate dense semantic rewards without requiring human annotations or additional training. This approach has been tested across five language tasks, demonstrating smoother reward improvements and more stable dynamics compared to traditional binary reward systems.

Read full article

via arXiv — cs.LG