arXiv:2511.08594v1 Announce Type: new 
Abstract: The ability of LLMs to represent diverse perspectives is critical as they increasingly impact society. However, recent studies reveal that alignment algorithms such as RLHF and DPO significantly reduce the diversity of LLM outputs. Not only do aligned LLMs generate text with repetitive structure and word choice, they also approach problems in more uniform ways, and their responses reflect a narrower range of societal perspectives. We attribute this problem to the KL divergence regularizer employed in preference learning algorithms. This causes the model to systematically overweight majority opinions and sacrifice diversity in its outputs. To address this, we propose Soft Preference Learning, which decouples the entropy and cross-entropy terms in the KL penalty - allowing for fine-grained control over LLM generation diversity. From a capabilities perspective, LLMs trained using Soft Preference Learning attain higher accuracy on difficult repeated sampling tasks and produce outputs with greater semantic and lexical diversity. From an alignment perspective, they are capable of representing a wider range of societal viewpoints and display improved logit calibration. Notably, Soft Preference Learning resembles, but is a Pareto improvement over, standard temperature scaling.

تسلط دراسة حديثة الضوء على أن خوارزميات المحاذاة مثل RLHF وDPO تقلل من تنوع مخرجات نماذج اللغة الكبيرة (LLMs). تُعزى هذه الانخفاضات في التنوع إلى مُنظم تباين KL، الذي يفضل الآراء السائدة. لمواجهة ذلك، يقترح المؤلفون التعلم الناعم للتفضيلات، الذي يحسن كل من الدقة والتنوع في مخرجات LLM، مما يسمح بتمثيل أوسع للوجهات النظر الاجتماعية.

Un estudio reciente destaca que los algoritmos de alineación como RLHF y DPO reducen la diversidad de las salidas de los modelos de lenguaje de gran tamaño (LLMs). Esta reducción en la diversidad se atribuye al regularizador de divergencia KL, que favorece las opiniones mayoritarias. Para contrarrestar esto, los autores proponen el Aprendizaje de Preferencias Suaves, que mejora tanto la precisión como la diversidad de las salidas de los LLM, permitiendo una representación más amplia de las perspectivas sociales.

Une étude récente souligne que les algorithmes d'alignement tels que RLHF et DPO réduisent la diversité des sorties des grands modèles de langage (LLMs). Cette réduction de la diversité est attribuée au régularisateur de divergence KL, qui favorise les opinions majoritaires. Pour contrer cela, les auteurs proposent l'apprentissage de préférences douces, qui améliore à la fois la précision et la diversité des sorties des LLM, permettant une représentation plus large des perspectives sociétales.

A recent study highlights that alignment algorithms like RLHF and DPO diminish the diversity of outputs from large language models (LLMs). This reduction in diversity is attributed to the KL divergence regularizer, which favors majority opinions. To counter this, the authors propose Soft Preference Learning, which enhances both the accuracy and diversity of LLM outputs, allowing for a broader representation of societal perspectives.

Diverse Preference Learning for Capabilities and Alignment

Was this article worth reading? Share it

Ready to build your own newsroom?