arXiv:2605.30201v1 Announce Type: new 
Abstract: We investigate a narrow but common failure mode of GRPO-style reinforcement learning in the context of sparse verifiable rewards: early updates contain more responses with negative advantages than those with positive advantages, while response-level length normalization ties the magnitude of the update to the length of the output. We propose Hysteretic Policy Optimization (HPO), a minimal modification of GRPO that reduces the weight of negative-advantage updates and replaces per-response length normalization with mean-length normalization. We further introduce Adaptive HPO (A-HPO), which sets the hysteretic weight based on batch-level advantage-sign statistics, thereby removing the need for tuning a fixed hysteretic weight. In our TeleLogs and Countdown experiments, A-HPO improves the reward per update compared to GRPO, with the largest gains in early sparse reward regimes. On TeleLogs, A-HPO achieves a final reward of 0.84, outperforming SAPO by 5%, GSPO by 11%, and GRPO by 15%, while maintaining a comparable response-length. On Countdown, A-HPO achieves the largest gains in initial and most difficult configurations across 1.5B-7B models. Ablation studies on the hysteretic weight show that the gains of A-HPO come from better balancing the contributions of positive and negative advantages compared to positive-only or fully symmetric updates.

تقدم دراسة جديدة تحسين السياسة الهيستيرية (HPO)، وهو تعديل على أسلوب التعلم المعزز GRPO، يهدف إلى معالجة المشكلات المتعلقة بالمكافآت القابلة للتحقق النادرة. تسلط الأبحاث الضوء على أن التحديثات المبكرة تحتوي غالبًا على مزيد من المزايا السلبية، مما يستدعي الحاجة إلى حل يقلل من وزن هذه التحديثات ويقوم بتطبيع أطوال الاستجابة.

Un nuevo estudio presenta la Optimización de Política Histerética (HPO), una modificación del aprendizaje por refuerzo estilo GRPO, destinada a abordar problemas relacionados con recompensas verificables escasas. La investigación destaca que las primeras actualizaciones a menudo contienen más ventajas negativas, lo que provoca la necesidad de una solución que reduzca el peso de estas actualizaciones y normalice las longitudes de respuesta.

Une nouvelle étude présente l'Optimisation de Politique Hystérique (HPO), une modification de l'apprentissage par renforcement de type GRPO, visant à résoudre des problèmes liés aux récompenses vérifiables rares. La recherche souligne que les premières mises à jour contiennent souvent plus d'avantages négatifs, ce qui nécessite une solution qui réduit le poids de ces mises à jour et normalise les longueurs de réponse.

A new study introduces Hysteretic Policy Optimization (HPO), a modification of GRPO-style reinforcement learning, aimed at addressing issues related to sparse verifiable rewards. The research highlights that early updates often contain more negative advantages, prompting the need for a solution that reduces the weight of these updates and normalizes response lengths.