arXiv:2606.12360v2 Announce Type: replace 
Abstract: Language-model post-training is the main stage at which model behavior is shaped, yet it still largely involves optimization of scalar rewards that summarize diverse desiderata. This abstraction gives practitioners little visibility into what their data actually teaches models, allowing spurious correlations to be learned by a model and inducing undesirable behaviors such as over-stylization and sycophancy. To address this problem, we ask: can we inspect a preference dataset before optimization and decide, at the level of concepts, which behaviors a model should be allowed to learn? Motivated by this, we introduce a data-centric post-training pipeline that uses interpretability protocols to develop statistical hypotheses for the latent concepts separating preferred from dispreferred generations, making them explicit for fine-grained user feedback. Building on this view, we unify several interpretability-based training protocols as ways of shaping rewards via feature or data interventions. Empirically, we show that our pipeline diagnoses undesirable signals in existing preference data, mitigates off-target learning, and can also help amplify or shape desired properties such as safeguards and model personality. More broadly, our results suggest that interpretability can turn post-training from optimizing opaque proxy rewards into a process of auditing and sculpting the learning signal itself.

دراسة حديثة حول ما بعد تدريب نماذج اللغة تسلط الضوء على أهمية القابلية للتفسير في تشكيل سلوك النموذج، حيث تكشف أن طرق التحسين الحالية غالبًا ما تخفي ما تعلمه البيانات للنماذج، مما يؤدي إلى سلوكيات غير مرغوب فيها. تقترح الدراسة خط أنابيب مركزي للبيانات يستخدم بروتوكولات القابلية للتفسير لتوضيح المفاهيم التي تميز المخرجات المفضلة عن غير المفضلة.

Un estudio reciente sobre el post-entrenamiento de modelos de lenguaje destaca la importancia de la interpretabilidad en la formación del comportamiento del modelo, revelando que los métodos de optimización actuales a menudo oscurecen lo que los datos enseñan a los modelos, lo que lleva a comportamientos indeseables. La investigación propone un pipeline centrado en los datos que utiliza protocolos de interpretabilidad para aclarar los conceptos que diferencian las salidas preferidas de las no preferidas.

Une étude récente sur le post-entraînement des modèles linguistiques souligne l'importance de l'interprétabilité dans la formation du comportement des modèles, révélant que les méthodes d'optimisation actuelles obscurcissent souvent ce que les données enseignent aux modèles, entraînant des comportements indésirables. La recherche propose un pipeline centré sur les données qui utilise des protocoles d'interprétabilité pour clarifier les concepts qui différencient les sorties préférées de celles non préférées.

A recent study on language-model post-training highlights the importance of interpretability in shaping model behavior, revealing that current optimization methods often obscure what data teaches models, leading to undesirable behaviors. The research proposes a data-centric pipeline that utilizes interpretability protocols to clarify the concepts that differentiate preferred from dispreferred model outputs.