arXiv:2606.00334v1 Announce Type: new 
Abstract: Various language domains have undergone remarkable changes in recent years; these shifts are largely attributed to the advent of Large Language Models and their misalignment with natural language usage. These misalignments are thought to partly originate in the preference-learning stage, e.g. Reinforcement Learning from Human Feedback, which generally makes models more useful but simultaneously may introduce systematic lexical bias. In terms of lexical behavior, this is visible in a model's preference for certain formats or the overuse of words (delve, furthermore), even when such patterns are not present in base model outputs. Research on lexical misalignment induced during preference training is constrained by reliance on manual curation. We address this, by introducing the Triangulated Preference Shift score, a metric that triangulates between human gold standards, base models, and instruct variants to isolate shifts induced specifically by preference learning, without manual curation. We provide data across six model families, anchor the results in the literature, and illustrate the general approach's utility by analyzing whether preference learning shifts models toward what could be interpreted as a "language of prestige". The metric provides an initial automated method to quantify behavioral shifts attributable to preference tuning, and thus, may help inform model alignment and development of trustworthy AI.

قدمت أبحاث جديدة مقياس تغيير التفضيل الثلاثي، وهو مقياس يهدف إلى عزل التحيز اللفظي في نماذج اللغة الكبيرة (LLMs) خلال مرحلة تعلم التفضيلات، وخاصة في التعلم المعزز من ردود الفعل البشرية. يسعى هذا المقياس إلى معالجة عدم التوافق بين مخرجات النموذج واستخدام اللغة الطبيعية، والذي تفاقم بسبب التحيزات النظامية التي تم إدخالها أثناء التدريب.

Una nueva investigación ha introducido el puntaje de cambio de preferencia triangulado, una métrica destinada a aislar el sesgo léxico en los Modelos de Lenguaje Grande (LLMs) durante la etapa de aprendizaje de preferencias, particularmente en el Aprendizaje por Refuerzo a partir de Retroalimentación Humana. Esta métrica busca abordar el desajuste entre las salidas del modelo y el uso del lenguaje natural, que ha sido exacerbado por sesgos sistemáticos introducidos durante el entrenamiento.

Une nouvelle recherche a introduit le score de décalage de préférence triangulé, une nouvelle métrique visant à isoler le biais lexical dans les grands modèles de langage (LLMs) lors de la phase d'apprentissage des préférences, en particulier dans l'apprentissage par renforcement à partir de retours humains. Cette métrique cherche à résoudre le désalignement entre les sorties des modèles et l'utilisation du langage naturel, exacerbée par les biais systémiques introduits lors de l'entraînement.

Recent research has introduced the Triangulated Preference Shift score, a new metric aimed at isolating lexical bias in Large Language Models (LLMs) during the preference-learning stage, particularly in Reinforcement Learning from Human Feedback. This metric seeks to address the misalignment between model outputs and natural language usage, which has been exacerbated by systematic biases introduced during training.