arXiv:2603.17145v2 Announce Type: replace 
Abstract: Large language models (LLMs) are increasingly deployed as automated evaluators that assign numeric scores to model outputs, a paradigm known as LLM-as-a-Judge. However, standard Reinforcement Learning (RL) methods typically rely on binary rewards (e.g., 0-1 accuracy), thereby ignoring the ordinal structure inherent in regression tasks; for instance, they fail to recognize that predicting 4 is significantly better than predicting 1 when the ground truth is 5. Conversely, existing regression-aware approaches are often confined to Supervised Fine-Tuning (SFT), limiting their ability to explore optimal reasoning paths. To bridge this gap, we propose \textbf{REAL} (\underline{RE}gression-\underline{A}ware Reinforcement \underline{L}earning), a principled RL framework designed to optimize regression rewards, and also proven to be optimal for correlation metrics. A key technical challenge is that the regression objective is explicitly policy-dependent, thus invalidating standard policy gradient methods. To address this, we employ the generalized policy gradient estimator, which naturally decomposes optimization into two complementary components: (1) exploration over Chain-of-Thought (CoT) trajectory, and (2) regression-aware prediction refinement of the final score. Extensive experiments across model scales (8B to 32B) demonstrate that REAL consistently outperforms both regression-aware SFT baselines and standard RL methods, exhibiting significantly better generalization on out-of-domain benchmarks. On Qwen3-32B specifically, we achieve gains of +8.40 Pearson and +7.20 Spearman correlation over the SFT baseline, and +18.30/+11.20 over the base model. These findings highlight the critical value of integrating regression objectives into RL exploration for accurate LLM evaluation.

تم اقتراح إطار عمل جديد يسمى REAL (التعلم المعزز القائم على الانحدار) لتحسين قدرات نماذج اللغة الكبيرة (LLMs) التي تعمل كقيمين آليين، حيث يعالج هذا الإطار القيود المفروضة على طرق التعلم المعزز التقليدية التي تعتمد على المكافآت الثنائية. يهدف هذا النهج إلى تحسين مكافآت الانحدار ويصمم لتحسين دقة تقييم مخرجات النماذج.

Se ha propuesto un nuevo marco llamado REAL (Aprendizaje por Refuerzo Sensible a la Regresión) para mejorar las capacidades de los grandes modelos de lenguaje (LLMs) que actúan como evaluadores automatizados, abordando las limitaciones de los métodos tradicionales de aprendizaje por refuerzo que dependen de recompensas binarias. Este enfoque optimiza las recompensas de regresión y está diseñado para mejorar la precisión de la evaluación de las salidas de los modelos.

Un nouveau cadre appelé REAL (Apprentissage par Renforcement Sensible à la Régression) a été proposé pour améliorer les capacités des grands modèles de langage (LLM) agissant en tant qu'évaluateurs automatisés, en abordant les limitations des méthodes d'apprentissage par renforcement traditionnelles qui reposent sur des récompenses binaires. Cette approche optimise les récompenses de régression et est conçue pour améliorer la précision d'évaluation des sorties des modèles.

A new framework called REAL (Regression-Aware Reinforcement Learning) has been proposed to enhance the capabilities of large language models (LLMs) acting as automated evaluators, addressing the limitations of traditional reinforcement learning methods that rely on binary rewards. This approach optimizes regression rewards and is designed to improve the evaluation accuracy of model outputs.