arXiv:2510.03520v2 Announce Type: replace 
Abstract: Ensuring safety is a foundational requirement for large language models (LLMs). Achieving an appropriate balance between enhancing the utility of model outputs and mitigating their potential for harm is a complex and persistent challenge. Contemporary approaches frequently formalize this problem within the framework of Constrained Markov Decision Processes (CMDPs) and employ established CMDP optimization techniques. However, these methods exhibit two notable limitations. First, their reliance on reward and cost functions renders performance highly sensitive to the underlying scoring mechanism, which must capture semantic meaning rather than being triggered by superficial keywords. Second, CMDP-based training entails tuning dual-variable, a process that is both computationally expensive and does not provide any provable safety guarantee for a fixed dual variable that can be exploitable through adversarial jailbreaks. To overcome these limitations, we introduce Certifiable Safe-RLHF (CS-RLHF) that introduces a cost model trained on a large-scale corpus to assign semantically grounded safety scores. In contrast to the lagrangian-based approach, CS-RLHF adopts a rectified penalty-based formulation. This design draws on the theory of exact penalty functions in constrained optimization, wherein constraint satisfaction is enforced directly through a suitably chosen penalty term. With an appropriately scaled penalty, feasibility of the safety constraints can be guaranteed at the optimizer, eliminating the need for dual-variable updates. Empirical evaluation demonstrates that CS-RLHF outperforms state-of-the-art LLM model responses rendering at-least 5 times efficient against nominal and jail-breaking prompts

دراسة حديثة تقدم نهجًا جديدًا لتعزيز أمان نماذج اللغة الكبيرة (LLMs) من خلال Certifiable Safe RLHF، الذي يركز على التأسيس الدلالي وتحسين قيود العقوبات الثابتة. تهدف هذه الطريقة إلى معالجة التحديات المستمرة في تحقيق التوازن بين فائدة النموذج وأمانه، خاصة في سياق عمليات اتخاذ القرار الماركوفية المقيدة (CMDPs).

Un estudio reciente presenta un enfoque novedoso para mejorar la seguridad de los grandes modelos de lenguaje (LLMs) a través de Certifiable Safe RLHF, que enfatiza el anclaje semántico y la optimización de restricciones de penalización fija. Este método busca abordar los desafíos persistentes de equilibrar la utilidad del modelo con la seguridad, particularmente en el contexto de los Procesos de Decisión de Markov Constrenidos (CMDPs).

Une étude récente présente une nouvelle approche pour améliorer la sécurité des grands modèles de langage (LLMs) grâce à Certifiable Safe RLHF, qui met l'accent sur le ancrage sémantique et l'optimisation des contraintes de pénalité fixe. Cette méthode vise à relever les défis persistants d'équilibrage de l'utilité du modèle avec la sécurité, en particulier dans le contexte des processus de décision markoviens contraints (CMDPs).

A recent study introduces a novel approach to enhancing the safety of large language models (LLMs) through Certifiable Safe RLHF, which emphasizes semantic grounding and fixed penalty constraint optimization. This method aims to address the persistent challenges of balancing model utility with safety, particularly in the context of Constrained Markov Decision Processes (CMDPs).