arXiv:2511.19504v1 Announce Type: cross 
Abstract: Reinforcement Learning from Human Feedback (RLHF) is widely used for aligning large language models, yet practitioners face a persistent puzzle: improving safety often reduces fairness, scaling to diverse populations becomes computationally intractable, and making systems robust often amplifies majority biases. We formalize this tension as the Alignment Trilemma: no RLHF system can simultaneously achieve (i) epsilon-representativeness across diverse human values, (ii) polynomial tractability in sample and compute complexity, and (iii) delta-robustness against adversarial perturbations and distribution shift. Through a complexity-theoretic analysis integrating statistical learning theory and robust optimization, we prove that achieving both representativeness (epsilon <= 0.01) and robustness (delta <= 0.001) for global-scale populations requires Omega(2^{d_context}) operations, which is super-polynomial in the context dimensionality. We show that current RLHF implementations resolve this trilemma by sacrificing representativeness: they collect only 10^3--10^4 samples from homogeneous annotator pools while 10^7--10^8 samples are needed for true global representation. Our framework provides a unified explanation for documented RLHF pathologies including preference collapse, sycophancy, and systematic bias amplification. We conclude with concrete directions for navigating these fundamental trade-offs through strategic relaxations of alignment requirements.

دراسة حديثة تُشكل مفهوم تريليما التوافق في التعلم المعزز من خلال التغذية الراجعة البشرية (RLHF)، مما يبرز الصراع المتأصل بين تحقيق التمثيل، والقدرة الحاسوبية، والموثوقية في أنظمة الذكاء الاصطناعي. تكشف التحليلات أن تحقيق كل من التمثيل والموثوقية للسكان العالميين يتطلب عمليات فوق متعددة الحدود، مما يعقد توافق نماذج اللغة الكبيرة.

Un estudio reciente formaliza el Trilema de Alineación en el Aprendizaje por Refuerzo a partir de Retroalimentación Humana (RLHF), destacando el conflicto inherente entre lograr representatividad, tractabilidad computacional y robustez en los sistemas de IA. El análisis revela que cumplir tanto con la representatividad como con la robustez para poblaciones globales requiere operaciones super-polinomiales, complicando la alineación de grandes modelos de lenguaje.

Une étude récente formalise le Trilemme d'Alignement dans l'Apprentissage par Renforcement à partir de Retours Humains (RLHF), mettant en évidence le conflit inhérent entre l'atteinte de la représentativité, la tractabilité computationnelle et la robustesse dans les systèmes d'IA. L'analyse révèle que répondre à la fois à la représentativité et à la robustesse pour des populations mondiales nécessite des opérations super-polynomiales, compliquant l'alignement des grands modèles de langage.

A recent study formalizes the Alignment Trilemma in Reinforcement Learning from Human Feedback (RLHF), highlighting the inherent conflict between achieving representativeness, computational tractability, and robustness in AI systems. The analysis reveals that meeting both representativeness and robustness for global populations requires super-polynomial operations, complicating the alignment of large language models.

Position: The Complexity of Perfect AI Alignment -- Formalizing the RLHF Trilemma

arXiv:2601.08100v1 Announce Type: new 
Abstract: Understanding the generalization behavior of deep neural networks remains a fundamental challenge in modern statistical learning theory. Among existing approaches, PAC-Bayesian norm-based bounds have demonstrated particular promise due to their data-dependent nature and their ability to capture algorithmic and geometric properties of learned models. However, most existing results rely on isotropic Gaussian posteriors, heavy use of spectral-norm concentration for weight perturbations, and largely architecture-agnostic analyses, which together limit both the tightness and practical relevance of the resulting bounds. To address these limitations, in this work, we propose a unified framework for PAC-Bayesian norm-based generalization by reformulating the derivation of generalization bounds as a stochastic optimization problem over anisotropic Gaussian posteriors. The key to our approach is a sensitivity matrix that quantifies the network outputs with respect to structured weight perturbations, enabling the explicit incorporation of heterogeneous parameter sensitivities and architectural structures. By imposing different structural assumptions on this sensitivity matrix, we derive a family of generalization bounds that recover several existing PAC-Bayesian results as special cases, while yielding bounds that are comparable to or tighter than state-of-the-art approaches. Such a unified framework provides a principled and flexible way for geometry-/structure-aware and interpretable generalization analysis in deep learning.

تشير دراسة جديدة إلى إطار موحد لـ PAC-Bayes للحدود العامة المعتمدة على المعايير، حيث تتناول التحديات المتعلقة بفهم سلوك التعميم في الشبكات العصبية العميقة. تعيد هذه الدراسة صياغة اشتقاق هذه الحدود كمشكلة تحسين عشوائي على توزيعات غاوسية غير متساوية، بهدف تعزيز الصلة العملية للنتائج.

Un nuevo estudio propone un marco PAC-Bayes unificado para los límites de generalización basados en normas, abordando los desafíos de comprender el comportamiento de generalización de las redes neuronales profundas. La investigación reformula la derivación de estos límites como un problema de optimización estocástica sobre posteriors gaussianos anisotrópicos, con el objetivo de mejorar la relevancia práctica de los resultados.

Une nouvelle étude propose un cadre PAC-Bayésien unifié pour les bornes de généralisation basées sur les normes, abordant les défis de la compréhension du comportement de généralisation des réseaux de neurones profonds. La recherche reformule la dérivation de ces bornes comme un problème d'optimisation stochastique sur des postérieurs gaussiens anisotropes, visant à améliorer la pertinence pratique des résultats.

A new study proposes a unified PAC-Bayesian framework for norm-based generalization bounds, addressing the challenges of understanding deep neural networks' generalization behavior. The research reformulates the derivation of these bounds as a stochastic optimization problem over anisotropic Gaussian posteriors, aiming to enhance the practical relevance of the results.

Towards A Unified PAC-Bayesian Framework for Norm-based Generalization Bounds

One More Thing in AI – Your Shortcut to AI Mastery

Position: The Complexity of Perfect AI Alignment -- Formalizing the RLHF Trilemma

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

LCW

Leiga

Linkjob AI

LangWatch

Ready to build your own newsroom?