arXiv:2604.18235v2 Announce Type: replace 
Abstract: Search agents achieve strong question-answering performance through multi-turn interactions with search engines, with Group Relative Policy Optimization (GRPO) being a widely used training algorithm. However, GRPO-style algorithms still face several challenges in multi-hop search settings. First, correct intermediate steps are often penalized when the final answer is wrong. Second, training is highly unstable, often causing degradation of natural language ability or even catastrophic training collapse. Our analysis attributes these issues to coarse-grained advantage assignment and an imbalance between positive and negative advantages. To address these problems, we propose CalibAdv, an advantage calibration method specifically designed for search agents that enables more accurate and more stable modeling of penalties and rewards. Specifically, CalibAdv leverages the correctness of intermediate steps to downscale excessive negative advantages at a fine-grained level. It then further rebalances positive and negative advantages to improve training stability. Importantly, CalibAdv adopts a lightweight design that calibrates advantages from standard rollout signals, making it simple and easy to deploy. Extensive experiments across three models and seven benchmarks demonstrate that CalibAdv improves both model performance and training stability. Our code is available at https://github.com/wujwyi/CalibAdv.

تسلط التطورات الأخيرة في تدريب وكلاء البحث الضوء على إدخال CalibAdv، وهي طريقة لمعايرة المزايا تهدف إلى تعزيز استقرار ودقة خوارزميات تحسين السياسات النسبية الجماعية (GRPO). تتناول هذه الطريقة التحديات مثل معاقبة الخطوات الوسيطة الصحيحة عندما تكون الإجابات النهائية غير صحيحة وعدم استقرار عمليات التدريب.

Recientes avances en la formación de agentes de búsqueda destacan la introducción de CalibAdv, un método de calibración de ventajas diseñado para mejorar la estabilidad y precisión de los algoritmos de Optimización de Políticas Relativas en Grupo (GRPO). Este método aborda desafíos como la penalización de pasos intermedios correctos cuando las respuestas finales son incorrectas y la inestabilidad de los procesos de entrenamiento.

Des avancées récentes dans la formation des agents de recherche mettent en lumière l'introduction de CalibAdv, une méthode de calibration des avantages conçue pour améliorer la stabilité et l'exactitude des algorithmes de Group Relative Policy Optimization (GRPO). Cette méthode aborde des défis tels que la pénalisation des étapes intermédiaires correctes lorsque les réponses finales sont incorrectes et l'instabilité des processus de formation.

Recent advancements in search agent training highlight the introduction of CalibAdv, an advantage calibration method designed to enhance the stability and accuracy of Group Relative Policy Optimization (GRPO) algorithms. This method addresses challenges such as penalizing correct intermediate steps when final answers are incorrect and the instability of training processes.