arXiv:2605.28150v1 Announce Type: new 
Abstract: Large scale reinforcement learning has become a central tool for improving reasoning in large language models. At this scale, generation is often lagged or asynchronous, so updates are performed on data collected by older policies. This makes learning inherently off-policy. Most existing approaches nevertheless remain rooted in PPO-style trust-region objectives, treating training as approximately on-policy and using importance weights to correct distribution mismatch. These corrections can introduce high variance, destabilize optimization, and accelerate entropy collapse. Recent work suggests an alternative: rather than correcting the mismatch, one can embrace off-policy data and remove importance weights, often yielding stronger algorithms. In this paper, we provide an intuitive construction of off-policy objectives that include successful off-policy objectives and show that their effectiveness can be understood through implicit pessimism: they optimize toward target policies that are more conservative than their nominal objectives suggest. This perspective explains why some particular implementation choices improve stability: they implicitly control the effective target distribution. We then propose a principled modification that stabilize this induced distribution and improve off-policy learning.

دراسة حديثة بعنوان 'Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think' تناقش تحديات التعلم المعزز على نطاق واسع، وخاصة كيف يمكن أن يعزز التعلم خارج السياسة التفكير في نماذج اللغة الكبيرة من خلال احتضان البيانات التي تم جمعها من سياسات أقدم دون الحاجة إلى أوزان الأهمية.

Un estudio reciente titulado 'Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think' discute los desafíos del aprendizaje por refuerzo a gran escala, particularmente cómo el aprendizaje fuera de política puede mejorar el razonamiento en grandes modelos de lenguaje al adoptar datos recopilados de políticas más antiguas sin la necesidad de pesos de importancia.

Une étude récente intitulée 'Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think' aborde les défis de l'apprentissage par renforcement à grande échelle, en particulier comment l'apprentissage hors politique peut améliorer le raisonnement dans les grands modèles de langage en adoptant des données collectées à partir de politiques plus anciennes sans avoir besoin de poids d'importance.

A recent study titled 'Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think' discusses the challenges of large-scale reinforcement learning, particularly how off-policy learning can enhance reasoning in large language models by embracing data collected from older policies without the need for importance weights.