arXiv:2512.22955v4 Announce Type: replace 
Abstract: Recent advancements have shown that reinforcement learning (RL) can substantially improve the reasoning abilities of large language models (LLMs). The effectiveness of such RL training, however, depends critically on the exploration space defined by the pre-trained model's token-output distribution. In this paper, we revisit the standard cross-entropy loss, interpreting it as a specific instance of policy gradient optimization applied within a single-step episode. To systematically study how the pre-trained distribution shapes the exploration potential for subsequent RL, we propose a generalized pre-training objective that adapts on-policy RL principles to supervised learning. By framing next-token prediction as a stochastic decision process, we introduce a reward-shaping strategy that explicitly balances diversity and precision. Our method employs a positive reward scaling factor to control probability concentration on ground-truth tokens and a rank-aware mechanism that treats high-ranking and low-ranking negative tokens asymmetrically. This allows us to reshape the pre-trained token-output distribution and investigate how to provide a more favorable exploration space for RL, ultimately enhancing end-to-end reasoning performance. Contrary to the intuition that higher distribution entropy facilitates effective exploration, we find that imposing a precision-oriented prior yields a superior exploration space for RL.

أظهرت التطورات الأخيرة في التعلم المعزز (RL) إمكانات كبيرة في تحسين قدرات التفكير لنماذج اللغة الكبيرة (LLMs). تعيد دراسة جديدة النظر في خسارة الانتروبيا المتقاطعة، حيث تفسرها كحالة معينة من تحسين تدرج السياسة، وتقترح هدف تدريب مسبق عام يدمج مبادئ RL على السياسة في التعلم الموجه، بهدف تحقيق توازن بين التنوع والدقة في التنبؤ بالرمز التالي.

Los avances recientes en el aprendizaje por refuerzo (RL) han mostrado un potencial significativo para mejorar las capacidades de razonamiento de los grandes modelos de lenguaje (LLMs). Un nuevo estudio revisita la pérdida de entropía cruzada, interpretándola como una instancia de optimización de gradiente de política, y propone un objetivo de preentrenamiento generalizado que integra principios de RL en línea en el aprendizaje supervisado, con el objetivo de equilibrar diversidad y precisión en la predicción del siguiente token.

Les avancées récentes en apprentissage par renforcement (RL) ont montré un potentiel significatif pour améliorer les capacités de raisonnement des grands modèles de langage (LLMs). Une nouvelle étude revisite la perte d'entropie croisée, l'interprétant comme une instance d'optimisation de gradient de politique, et propose un objectif de pré-entraînement généralisé qui intègre les principes de RL en ligne dans l'apprentissage supervisé, visant à équilibrer diversité et précision dans la prédiction du prochain token.

Recent advancements in reinforcement learning (RL) have shown significant potential in enhancing the reasoning capabilities of large language models (LLMs). A new study revisits the cross-entropy loss, interpreting it as a policy gradient optimization instance, and proposes a generalized pre-training objective that integrates on-policy RL principles into supervised learning, aiming to balance diversity and precision in next-token prediction.