arXiv:2606.11046v1 Announce Type: new 
Abstract: Instruction-tuned LLMs are increasingly converted into reasoning models through post-training to improve multi-step task performance. This conversion is usually optimized for reasoning accuracy, without explicitly preserving the alignment behavior of the instruction-tuned model, such as safe refusal, bias avoidance, and privacy protection. We ask: does this conversion preserve alignment? We study this question through a trustworthiness audit and find that it is not behavior-preserving by default. For a systematic analysis, we compare reasoning models produced via supervised fine-tuning, RL-based post-training, and distillation against matched instruction-tuned baselines across six trustworthiness dimensions: safety, toxicity, stereotyping and bias, machine ethics, privacy, and out-of-distribution robustness. We observe that reasoning models often improve on reasoning benchmarks but exhibit alignment regressions, including increased toxicity, amplified stereotyping, miscalibrated refusal, and contextual privacy leakage. These regressions are consistent with behavioral drift from the instruction-tuned baseline, measured by KL divergence. Overall, our results point to the broader conclusion that trustworthiness metrics are essential for evaluating reasoning models and should be reported alongside gains in reasoning capability.

دراسة حديثة نُشرت على arXiv تبحث في موثوقية نماذج التفكير الكبيرة (LRMs) المستمدة من نماذج اللغة الكبيرة (LLMs) المعدلة وفقًا للتعليمات. تكشف الأبحاث أنه على الرغم من أن هذه النماذج غالبًا ما تعزز دقة التفكير، إلا أنها لا تحافظ بشكل تلقائي على سلوكيات التوافق مثل الأمان وتجنب التحيز، مما يؤدي إلى تراجع في التوافق.

Un estudio reciente publicado en arXiv investiga la confiabilidad de los grandes modelos de razonamiento (LRMs) derivados de modelos de lenguaje de gran tamaño (LLMs) ajustados por instrucciones. La investigación revela que, aunque estos modelos a menudo mejoran la precisión del razonamiento, no preservan inherentemente los comportamientos de alineación, como la seguridad y la evitación de sesgos, lo que conduce a regresiones en la alineación.

Une étude récente publiée sur arXiv examine la fiabilité des grands modèles de raisonnement (LRMs) dérivés de modèles de langage de grande taille (LLMs) ajustés par instruction. La recherche révèle que, bien que ces modèles améliorent souvent la précision du raisonnement, ils ne préservent pas intrinsèquement les comportements d'alignement tels que la sécurité et l'évitement des biais, entraînant des régressions d'alignement.

A recent study published on arXiv investigates the trustworthiness of large reasoning models (LRMs) derived from instruction-tuned large language models (LLMs). The research reveals that while these models often enhance reasoning accuracy, they do not inherently preserve alignment behaviors such as safety and bias avoidance, leading to alignment regressions.