arXiv:2605.26738v1 Announce Type: new 
Abstract: Human communication depends on implicit social signals where effectiveness is shaped by tone, context, and conversational norms rather than semantic content alone. We introduce KARMA (Karma-Aligned Reward Model Adaptation), a framework for LLM learning of context-sensitive conversational behavior from large-scale social interaction data. KARMA trains a reward model on Reddit conversations to predict response valuation conditioned on context, and uses this signal to fine-tune language models via reinforcement learning to improve performance on pragmatics-mediated tasks. Critically, we find that the highest performing reward model does not lead to better downstream model alignment: a reward model relying exclusively on conversational context was a worse predictor of Reddit karma but yielded substantially better downstream performance. We evaluate the effects of KARMA applied to a downstream model with and without direct exposure to the social media data. The resulting models show improved pragmatics-mediated behaviors with largely mitigated undesirable side effects. Factuality is consistently diminished by KARMA across all conditions, including when the downstream model has no direct exposure to Reddit data, suggesting that this tension is embedded in the reward signal itself rather than introduced by noisy training data.

يمثل تقديم نموذج KARMA (تكييف نموذج المكافأة المتوافق مع الكارما) تقدمًا كبيرًا في تدريب نماذج اللغة الكبيرة (LLMs) من خلال استخدام سلوك المحادثة الحساس للسياق المستمد من بيانات التفاعل الاجتماعي الواسعة على منصات مثل Reddit. يهدف هذا الإطار إلى تحسين فعالية نماذج اللغة في فهم والاستجابة للإشارات الاجتماعية الدقيقة بما يتجاوز المحتوى الدلالي البسيط.

La introducción de KARMA (Karma-Aligned Reward Model Adaptation) marca un avance significativo en el entrenamiento de modelos de lenguaje de gran tamaño (LLMs) al utilizar un comportamiento conversacional sensible al contexto derivado de datos de interacción social extensos en plataformas como Reddit. Este marco tiene como objetivo mejorar la efectividad de los LLMs en la comprensión y respuesta a señales sociales matizadas más allá del contenido semántico puro.

L'introduction de KARMA (Karma-Aligned Reward Model Adaptation) représente une avancée significative dans la formation des modèles de langage de grande taille (LLMs) en utilisant un comportement conversationnel sensible au contexte dérivé de données d'interaction sociale étendues sur des plateformes comme Reddit. Ce cadre vise à améliorer l'efficacité des LLMs dans la compréhension et la réponse à des signaux sociaux nuancés au-delà du simple contenu sémantique.

The introduction of KARMA (Karma-Aligned Reward Model Adaptation) marks a significant advancement in the training of large language models (LLMs) by utilizing context-sensitive conversational behavior derived from extensive social interaction data on platforms like Reddit. This framework aims to enhance the effectiveness of LLMs in understanding and responding to nuanced social signals beyond mere semantic content.

KARMA: Karma-Aligned Reward Model Adaptation

arXiv:2601.10896v2 Announce Type: replace 
Abstract: LLMs are increasingly used as third-party judges, yet their reliability when evaluating speakers in dialogue remains poorly understood. We show that LLMs judge identical claims differently depending on framing: the same content receives different verdicts when presented as a statement to verify ("Is this statement correct?") versus attributed to a speaker ("Is this speaker correct?"). We call this dialogic deference and introduce DialDefer, a framework for detecting and mitigating these framing-induced judgment shifts. Our Dialogic Deference Score (DDS) captures directional shifts that aggregate accuracy obscures. Across ten domains, 3k+ instances, and five models, conversational framing induces large shifts (mean|DDS|=15.9 percentage points (pp) across models, p < .0001) while accuracy remains stable (<2 pp), with effects amplifying 2--5x on naturalistic Reddit conversations. This effect is domain-dependent: a single model can shift toward disagreement (skepticism) on graduate-level science and toward agreement (deference) on social judgment. Ablations reveal that human-vs-LLM attribution drives the largest shifts (17.7 pp swing), suggesting models treat disagreement with humans as more costly than with AI. Mitigation attempts can reduce deference but over-correct into skepticism, revealing a calibration problem beyond accuracy optimization.

قدمت دراسة حديثة إطار عمل يسمى DialDefer، مصمم للكشف عن التفضيل الحواري والتخفيف منه في نماذج اللغة الكبيرة (LLMs)، كاشفة أن هذه النماذج يمكن أن تحكم على ادعاءات متطابقة بشكل مختلف بناءً على كيفية صياغتها. وجدت الأبحاث تغييرات كبيرة في الحكم اعتمادًا على ما إذا كانت الادعاءات مقدمة كبيانات أو منسوبة إلى متحدثين، مع متوسط درجة تفضيل حواري تشير إلى تغيير متوسط قدره 15.9 نقطة مئوية عبر نماذج مختلفة.

Un estudio reciente presentó DialDefer, un marco diseñado para detectar y mitigar la deferencia dialógica en los modelos de lenguaje de gran tamaño (LLMs), revelando que estos modelos pueden juzgar afirmaciones idénticas de manera diferente según cómo se enmarquen. La investigación encontró cambios significativos en el juicio dependiendo de si las afirmaciones se presentaban como declaraciones o se atribuían a hablantes, con un puntaje medio de deferencia dialógica que indica un cambio promedio de 15.9 puntos porcentuales entre varios modelos.

Une étude récente a présenté DialDefer, un cadre conçu pour détecter et atténuer la déférence dialogique dans les modèles de langage de grande taille (LLMs), révélant que ces modèles peuvent juger des affirmations identiques différemment selon la manière dont elles sont formulées. La recherche a trouvé des changements significatifs dans le jugement en fonction de la présentation des affirmations, avec un score moyen de déférence dialogique indiquant un changement moyen de 15,9 points de pourcentage à travers divers modèles.

A recent study introduced DialDefer, a framework designed to detect and mitigate dialogic deference in large language models (LLMs), revealing that LLMs can judge identical claims differently based on how they are framed. The research found significant shifts in judgment depending on whether claims were presented as statements or attributed to speakers, with an average Dialogic Deference Score indicating a mean shift of 15.9 percentage points across various models.

KARMA: Karma-Aligned Reward Model Adaptation

Was this article worth reading? Share it

Ready to build your own newsroom?