Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs

arXiv — cs.LGMonday, December 22, 2025 at 5:00:00 AM
  • A new framework called Turn-PPO has been introduced to enhance multi-turn reinforcement learning (RL) for interactive large language models (LLMs). This approach addresses the limitations of the Group Relative Policy Optimization (GRPO) algorithm, particularly in long-horizon reasoning tasks, by employing a turn-level Markov Decision Process (MDP) formulation. The effectiveness of Turn-PPO has been demonstrated through experiments on the WebShop and Sokoban datasets.
  • The development of Turn-PPO is significant as it offers a more robust alternative to existing RL strategies, potentially improving the performance of LLMs in complex, interactive environments. By focusing on turn-level advantage estimation, this framework aims to enhance the decision-making capabilities of LLM agents, making them more effective in real-world applications.
  • This advancement in RL for LLMs reflects a broader trend towards improving multi-agent systems and collaborative learning frameworks. As researchers explore various methodologies, including contextualization of web pages and strategic decision-making in gaming scenarios, the integration of more sophisticated RL techniques like Turn-PPO may pave the way for more capable and adaptable AI systems across diverse applications.
— via World Pulse Now AI Editorial System

Was this article worth reading? Share it

Recommended apps based on your readingExplore all apps
Continue Readings
Attention Projection Mixing and Exogenous Anchors
NeutralArtificial Intelligence
A new study introduces ExoFormer, a transformer model that utilizes exogenous anchor projections to enhance attention mechanisms, addressing the challenge of balancing stability and computational efficiency in deep learning architectures. This model demonstrates improved performance metrics, including a notable increase in downstream accuracy and data efficiency compared to traditional internal-anchor transformers.
SwiftMem: Fast Agentic Memory via Query-aware Indexing
PositiveArtificial Intelligence
SwiftMem has been introduced as a query-aware agentic memory system designed to enhance the efficiency of large language model (LLM) agents by enabling sub-linear retrieval through specialized indexing techniques. This system addresses the limitations of existing memory frameworks that rely on exhaustive retrieval methods, which can lead to significant latency issues as memory storage expands.
User-Oriented Multi-Turn Dialogue Generation with Tool Use at scale
NeutralArtificial Intelligence
A new framework for user-oriented multi-turn dialogue generation has been developed, leveraging large reasoning models (LRMs) to create dynamic, domain-specific tools for task completion. This approach addresses the limitations of existing datasets that rely on static toolsets, enhancing the interaction quality in human-agent collaborations.
Detecting Mental Manipulation in Speech via Synthetic Multi-Speaker Dialogue
NeutralArtificial Intelligence
A new study has introduced the SPEECHMENTALMANIP benchmark, marking the first exploration of mental manipulation detection in spoken dialogues, utilizing synthetic multi-speaker audio to enhance a text-based dataset. This research highlights the challenges of identifying manipulative speech tactics, revealing that models trained on audio exhibit lower recall compared to text.
RULERS: Locked Rubrics and Evidence-Anchored Scoring for Robust LLM Evaluation
PositiveArtificial Intelligence
The recent introduction of RULERS (Rubric Unification, Locking, and Evidence-anchored Robust Scoring) addresses challenges in evaluating large language models (LLMs) by transforming natural language rubrics into executable specifications, thereby enhancing the reliability of assessments.
PrivGemo: Privacy-Preserving Dual-Tower Graph Retrieval for Empowering LLM Reasoning with Memory Augmentation
PositiveArtificial Intelligence
PrivGemo has been introduced as a privacy-preserving framework designed for knowledge graph (KG)-grounded reasoning, addressing the risks associated with using private KGs in large language models (LLMs). This dual-tower architecture maintains local knowledge while allowing remote reasoning through an anonymized interface, effectively mitigating semantic and structural exposure.
Rescind: Countering Image Misconduct in Biomedical Publications with Vision-Language and State-Space Modeling
PositiveArtificial Intelligence
A new framework named Rescind has been introduced to combat image manipulation in biomedical publications, addressing the challenges of detecting forgeries that arise from domain-specific artifacts and complex textures. This framework combines vision-language prompting with state-space modeling to enhance the detection and generation of biomedical image forgeries.
Whose Facts Win? LLM Source Preferences under Knowledge Conflicts
NeutralArtificial Intelligence
A recent study examined the preferences of large language models (LLMs) in resolving knowledge conflicts, revealing a tendency to favor information from credible sources like government and newspaper outlets over social media. This research utilized a novel framework to analyze how these source preferences influence LLM outputs.

Ready to build your own newsroom?

Subscribe to unlock a personalised feed, podcasts, newsletters, and notifications tailored to the topics you actually care about