Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoning

Tree-OPO introduces a groundbreaking method for optimizing policies in preference-based reinforcement learning by leveraging Monte Carlo Tree Search (MCTS) to generate high-quality intermediate trajectories. This approach redefines Group Relative Policy Optimization (GRPO) through a staged training paradigm, utilizing MCTS rollouts to create a structured curriculum. The introduction of Staged Advantage Estimation (SAE) addresses the challenge of calculating advantages for samples from various prefixes, each with unique expected returns. Empirical results demonstrate that SAE not only improves final accuracy over standard GRPO but also enhances sample efficiency by reducing gradient variance. This innovation is crucial for advancing the capabilities of large language models in complex reasoning tasks, particularly in mathematical and symbolic domains.