Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning
A new framework called Plan-Then-Action Enhanced Reasoning with Group Relative Policy Optimization (PTA-GRPO) has been proposed to improve the reasoning capabilities of Large Language Models (LLMs) by addressing the limitations of local decision-making in token-level generation. This two-stage approach enhances high-level planning and fine-grained reasoning through supervised fine-tuning and reinforcement learning methods.
WPN Brief
- What Happened
A new framework called Plan-Then-Action Enhanced Reasoning with Group Relative Policy Optimization (PTA-GRPO) has been proposed to improve the reasoning capabilities of Large Language Models (LLMs) by addressing the limitations of local decision-making in token-level generation. This two-stage approach enhances high-level planning and fine-grained reasoning through supervised fine-tuning and reinforcement learning methods.
- Why It Matters
The development of PTA-GRPO is significant as it aims to reduce computational costs associated with existing reasoning methods while enhancing the reliability of reasoning trajectories in LLMs. This advancement could lead to more efficient applications of LLMs across various domains, improving their overall performance.
- The Bigger Picture
This initiative reflects a broader trend in artificial intelligence research, where enhancing reasoning capabilities in LLMs is a key focus. Various approaches, including reinforcement learning and hybrid training methods, are being explored to tackle the challenges of reasoning accuracy and efficiency, indicating a growing recognition of the need for improved frameworks in AI reasoning.
Related Reports
More coverage on this story
10 reports across the wire
Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression
A novel framework named Extra-CoT has been introduced to enhance the efficiency of Large Language Models (LLMs) by implementing Extreme-Ratio Chain-of-Thought Compression. This method aims to significantly reduce computational overhead during inference while maintaining high logical fidelity and answer accuracy, addressing the limitations of existing compression techniques.
Embodied Task Planning via Graph-Informed Action Generation with Large Language Models
A new framework named GiG has been proposed to enhance the planning capabilities of Large Language Models (LLMs) in embodied tasks. This framework utilizes a Graph-in-Graph architecture to structure memory and improve the coherence of long-horizon planning, addressing limitations faced by standard LLM planners in dynamic environments.
PathWise: Planning through World Model for Automated Heuristic Design via Self-Evolving LLMs
A novel framework named PathWise has been introduced, leveraging Large Language Models (LLMs) to enhance automated heuristic design (AHD) for combinatorial optimization problems (COPs). This approach formulates heuristic generation as a sequential decision-making process, utilizing an entailment graph to maintain a compact memory of the search trajectory, thereby improving the efficiency and effectiveness of heuristic generation.
Hybrid Training for Vision-Language-Action Models
Recent advancements in artificial intelligence have led to the exploration of Hybrid Training (HyT) for Vision-Language-Action (VLA) models, which aims to enhance performance by allowing these models to learn from intermediate thoughts before executing actions. This approach addresses the challenge of inference time, which can be negatively impacted by longer chains of thought.
Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Models
A new approach called Multi-domain Contrastive Policy Optimization (MCPO) has been proposed to enhance the reasoning capabilities of Large Reasoning Models (LRMs) by promoting cross-domain knowledge sharing and in-domain knowledge consolidation, addressing limitations in existing Group Relative Policy Optimization (GRPO) methods.
Reason--Imagine--Act: Closed-Loop LLM Decision Making with World Models for Autonomous Driving
arXiv:2605.24004v1 Announce Type: cross Abstract: Large language models (LLMs) are promising for autonomous driving, but semantics-only decision policies can yield physically unsafe behavior in dynamic traffic. Existing methods either perform online language reasoning without explicit dynamics verification or use world models mainly in offline pipelines, leaving a gap between semantic intent and physical feasibility at decision time. We propose Reason--Imagine--Act (RIA), a closed-loop framework that couples an LLM reasoner with an action-conditioned world model for online safety verification. At each step, the LLM proposes an action template and candidate sub-actions, the world model performs short-horizon rollouts, and a safety scorer selects the safest executable action with feedback to the next reasoning step. Under a unified CARLA point-goal protocol (1000 episodes), RIA achieves 80.05% route completion, 51.10% arrival rate, and 0.20% collision rate. Under the same closed-loop interface, RIA consistently outperforms training-free baselines, including CARLA TM and MADA, on core closed-loop metrics. For reproducibility, code is available at https://github.com/pku-smart-city/source_code/tree/main/RIA.
Tracing the ongoing emergence of human-like reasoning in Large Language Models
A recent study has explored the emergence of human-like reasoning in Large Language Models (LLMs), revealing that while these models exhibit impressive performance across various tasks, their reasoning processes differ significantly from human reasoning. The research involved a population-matching experiment comparing the conditional inference capabilities of 25 LLMs with those of an equal number of human participants across four languages.
STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning
The recent introduction of STRIDE, a novel training framework for Large Language Models (LLMs), shifts the focus from scalar rewards to learnable stepwise language feedback, enhancing the reasoning capabilities of these models. This approach aims to overcome the limitations of costly annotations and information bottlenecks associated with traditional reinforcement learning methods.
From Reasoning to Code: GRPO Optimization for Underrepresented Languages
arXiv:2506.11027v3 Announce Type: replace Abstract: Generating accurate and executable code using Large Language Models (LLMs) remains a significant challenge for underrepresented programming languages, such as Prolog and Lisp, due to the scarcity of public training data compared to high-resource languages like Python. This paper introduces a generalizable Reinforcement Learning (RL) approach that combines small-scale versions of the Qwen2.5-Coder model with Group Relative Policy Optimization (GRPO) to enable effective code generation through reasoning. To address the limitations of sparse datasets, we integrate execution-driven feedback directly into the RL loop, utilizing a reward system that exploits both logical correctness and structural formatting. Experimental results on GSM8K dataset demonstrate significant improvements in reasoning quality and code accuracy across underrepresented languages. These findings underscore the potential of our approach to benefit a wide range of programming languages lacking extensive training resources by leveraging symbolic reasoning and interpreter-based feedback.
Taming the Thinker: Conditional Entropy Shaping for Adaptive LLM Reasoning
A new framework called Conditional Entropy Shaping (CES) has been introduced to enhance the reasoning capabilities of Large Language Models (LLMs) by dynamically controlling token-level response entropy. This approach allows LLMs to provide concise solutions for simple problems while promoting deeper exploration for more complex issues, implemented on the DeepSeek-R1-Distill-7B model and evaluated across 12 mathematical benchmarks.