arXiv:2606.03189v2 Announce Type: replace 
Abstract: Large Language Models (LLMs) as judges across various scenarios such as assessing model responses is becoming an increasingly accepted paradigm. However, existing judgment approaches often rely on trained judgers using fixed preference data, which tend to overlook diverse user preferences and struggle to adapt to real-world human-AI dialogue scenarios. To address these limitations, we propose SenseJudge, a customizable judgment framework driven by human preferences and SenseBench, a diverse and challenging instruction-following benchmark derived from real-world multi-turn interactions. We applied the automatic judgment framework and benchmark to two tasks: (1) LLMs as personalized judges, and (2) model ranking. We conducted extensive experiments, and the results demonstrate that the SenseJudge framework surpasses other judgment methods and models in the LLMs-as-personalized-judges task and achieves model ranking that aligns with real human sense. Additionally, we conducted analyses on position bias and consistency, alongside ablation studies, which affirmed the robustness of SenseJudge.

تم تقديم إطار العمل SenseJudge كنظام حكم يركز على الإنسان ويعتمد على التفضيلات، مما يعزز تقييم نماذج اللغة الكبيرة (LLMs) في سيناريوهات متنوعة، ويعالج قيود طرق الحكم التقليدية التي غالبًا ما تفشل في التقاط تنوع تفضيلات المستخدمين.

Se ha presentado el marco SenseJudge como un sistema de juicio centrado en el ser humano y basado en preferencias, que mejora la evaluación de los Modelos de Lenguaje de Gran Escala (LLMs) en diversos escenarios, abordando las limitaciones de los métodos de juicio tradicionales que a menudo no logran capturar la diversidad de preferencias de los usuarios.

Le cadre SenseJudge a été introduit comme un système de jugement centré sur l'humain et axé sur les préférences, qui améliore l'évaluation des modèles de langage de grande taille (LLMs) dans divers scénarios, en répondant aux limites des méthodes de jugement traditionnelles qui échouent souvent à capturer la diversité des préférences des utilisateurs.

The SenseJudge framework has been introduced as a human-centric, preference-driven judgment system that enhances the evaluation of Large Language Models (LLMs) in various scenarios, addressing the limitations of traditional judgment methods that often fail to capture diverse user preferences.