arXiv:2511.12596v1 Announce Type: cross 
Abstract: Large Language Models (LLMs) often suffer from mode collapse, repeatedly generating the same few completions even when many valid answers exist, limiting their diversity across a wide range of tasks. We introduce Group-Aware Policy Optimization (GAPO), a simple extension of the recent and popular Group Relative Policy Optimization (GRPO) that computes rewards over the group as a whole. GAPO enables learning from the group-level properties such as diversity and coverage. We demonstrate GAPO using a frequency-aware reward function that encourages uniform sampling over valid LLM completions, and show that GAPO-trained models produce valid and more diverse model responses. Beyond this setup, GAPO generalizes to open-ended prompts and improves response diversity without compromising accuracy on standard LLM benchmarks (GSM8K, MATH, HumanEval, MMLU-Pro). Our code will be made publicly available.

تعاني النماذج اللغوية الكبيرة (LLMs) غالبًا من انهيار الوضع، حيث تنتج استجابات محدودة على الرغم من وجود إجابات متنوعة. لمعالجة هذه المشكلة، تم تقديم تحسين السياسة الواعية بالمجموعات (GAPO)، وهو امتداد لتحسين السياسة النسبية للمجموعات (GRPO). يركز GAPO على الخصائص على مستوى المجموعة مثل التنوع والتغطية، باستخدام دالة مكافأة واعية بالتردد لتعزيز أخذ عينات موحدة من الاستجابات الصالحة. تشير النتائج إلى أن النماذج المدربة باستخدام GAPO تنتج استجابات أكثر تنوعًا وصلاحية مع الحفاظ على الدقة في المعايير القياسية مثل GSM8K وMATH وHumanEval وMMLU-Pro. سيتم توفير الشيفرة للجمهور.

Los Modelos de Lenguaje Grande (LLMs) a menudo sufren de colapso de modo, generando respuestas limitadas a pesar de la existencia de respuestas diversas. Para abordar este problema, se ha introducido la Optimización de Políticas Consciente de Grupos (GAPO), una extensión de la Optimización de Políticas Relativas a Grupos (GRPO). GAPO se centra en propiedades a nivel de grupo como la diversidad y la cobertura, utilizando una función de recompensa consciente de la frecuencia para fomentar un muestreo uniforme de las completaciones válidas. Los resultados indican que los modelos entrenados con GA…

Les grands modèles de langage (LLM) souffrent souvent d'un effondrement des modes, générant des réponses limitées malgré la diversité des réponses possibles. Pour remédier à ce problème, des chercheurs ont introduit l'optimisation de politique consciente des groupes (GAPO), une extension de l'optimisation de politique relative aux groupes (GRPO). GAPO se concentre sur les propriétés de groupe telles que la diversité et la couverture, utilisant une fonction de récompense sensible à la fréquence pour promouvoir un échantillonnage uniforme des complétions valides. Les résultats montrent que les m…

Large Language Models (LLMs) often experience mode collapse, generating limited responses despite the availability of diverse answers. To address this issue, researchers have introduced Group-Aware Policy Optimization (GAPO), an extension of Group Relative Policy Optimization (GRPO). GAPO focuses on group-level properties such as diversity and coverage, utilizing a frequency-aware reward function to promote uniform sampling of valid completions. The results indicate that models trained with GAPO yield more varied and valid responses while maintaining accuracy across standard benchmarks like GS…

Group-Aware Reinforcement Learning for Output Diversity in Large Language Models

arXiv:2601.08489v1 Announce Type: new 
Abstract: Safety-aligned language models systematically refuse harmful requests. While activation steering can modulate refusal, ablating the raw "refusal vector" calculated from contrastive harmful and harmless prompts often causes collateral damage and distribution drift. We argue this degradation occurs because the raw vector is polysemantic, entangling the refusal signal with core capability circuits and linguistic style.
  We introduce Surgical Refusal Ablation (SRA) to distill these steering directions. SRA constructs a registry of independent Concept Atoms representing protected capabilities and stylistic confounds, then uses ridge-regularized spectral residualization to orthogonalize the refusal vector against these directions. This yields a clean refusal direction that targets refusal-relevant structure while minimizing disruption to the model's semantic geometry.
  Across five models (Qwen3-VL and Ministral series), SRA achieves deep refusal reduction (0-2%) with negligible perplexity impact on Wikitext-2 (mean delta PPL approx. 0.02) and minimal distribution drift. Notably, standard ablation on Qwen3-VL-4B induces severe drift (first-token KL = 2.088), whereas SRA maintains the original distribution (KL = 0.044) while achieving the same 0% refusal rate. Using teacher-forced perplexity on GSM8K and MBPP as a high-resolution capability proxy, we show SRA preserves math and code distributions. These results suggest that common "model damage" is often "Ghost Noise," defined as the spectral bleeding of the dirty refusal direction into capability subspaces.

تهدف مقدمة عملية الاستئصال الجراحي للرفض (SRA) إلى تعزيز أمان نماذج اللغة من خلال تحسين قدراتها على الرفض، مما يقلل من الأضرار الجانبية والانحراف في التوزيع الناتج عن الأساليب التقليدية. تحقق SRA ذلك من خلال إنشاء سجل من الذرات المفاهيمية المستقلة واستخدام التصفية الطيفية المنتظمة بواسطة ridge لإنتاج اتجاه رفض واضح.

La introducción de la Ablación Quirúrgica de Rechazo (SRA) tiene como objetivo mejorar la seguridad de los modelos de lenguaje al refinar sus capacidades de rechazo, minimizando los daños colaterales y la deriva de distribución causados por métodos tradicionales. La SRA logra esto creando un registro de Átomos de Concepto independientes y utilizando la residualización espectral regularizada por ridge para producir una dirección de rechazo clara.

L'introduction de l'Ablation de Refus Chirurgical (SRA) vise à améliorer la sécurité des modèles de langage en affinant leurs capacités de refus, minimisant les dommages collatéraux et la dérive de distribution causés par les méthodes traditionnelles. Le SRA y parvient en créant un registre d'Atomes de Concept indépendants et en utilisant la résidualisation spectrale régularisée par crête pour produire une direction de refus claire.

The introduction of Surgical Refusal Ablation (SRA) aims to enhance the safety of language models by refining their refusal capabilities, minimizing collateral damage and distribution drift caused by traditional methods. SRA achieves this by creating a registry of independent Concept Atoms and utilizing ridge-regularized spectral residualization to produce a clean refusal direction.

Surgical Refusal Ablation: Disentangling Safety from Intelligence via Concept-Guided Spectral Cleaning

arXiv:2601.08343v1 Announce Type: cross 
Abstract: Multi-agent LLM systems routinely generate multiple candidate responses that are aggregated by an LLM judge. To reduce the dominant prefill cost in such pipelines, recent work advocates KV cache reuse across partially shared contexts and reports substantial speedups for generation agents. In this work, we show that these efficiency gains do not transfer uniformly to judge-centric inference. Across GSM8K, MMLU, and HumanEval, we find that reuse strategies that are effective for execution agents can severely perturb judge behavior: end-task accuracy may appear stable, yet the judge's selection becomes highly inconsistent with dense prefill. We quantify this risk using Judge Consistency Rate (JCR) and provide diagnostics showing that reuse systematically weakens cross-candidate attention, especially for later candidate blocks. Our ablation further demonstrates that explicit cross-candidate interaction is crucial for preserving dense-prefill decisions. Overall, our results identify a previously overlooked failure mode of KV cache reuse and highlight judge-centric inference as a distinct regime that demands dedicated, risk-aware system design.

تسلط الأبحاث الحديثة الضوء على أنه بينما يمكن أن تعزز إعادة استخدام ذاكرة التخزين المؤقت KV الكفاءة في أنظمة نماذج اللغة متعددة الوكلاء (LLM)، إلا أنها قد تؤثر سلبًا على أداء القضاة LLM، مما يؤدي إلى سلوكيات اختيار غير متسقة على الرغم من دقة المهام النهائية المستقرة.

Investigaciones recientes destacan que, aunque la reutilización de caché KV puede mejorar la eficiencia en sistemas de modelos de lenguaje multiagente (LLM), puede afectar negativamente el rendimiento de los jueces LLM, llevando a comportamientos de selección inconsistentes a pesar de una precisión estable en la tarea final.

Des recherches récentes soulignent que bien que la réutilisation du cache KV puisse améliorer l'efficacité des systèmes de modèles de langage multi-agents (LLM), elle peut nuire à la performance des juges LLM, entraînant des comportements de sélection incohérents malgré une précision stable des tâches finales.

Recent research highlights that while KV cache reuse can enhance efficiency in multi-agent large language model (LLM) systems, it can negatively impact the performance of LLM judges, leading to inconsistent selection behaviors despite stable end-task accuracy.

When KV Cache Reuse Fails in Multi-Agent Systems: Cross-Candidate Interaction is Crucial for LLM Judges

arXiv:2601.08427v1 Announce Type: cross 
Abstract: Group Relative Policy Optimization (GRPO) significantly enhances the reasoning performance of Large Language Models (LLMs). However, this success heavily relies on expensive external verifiers or human rules. Such dependency not only leads to significant computational costs and training latency, but also yields sparse rewards that hinder optimization efficiency. To address these challenges, we propose Latent-GRPO, a framework that derives intrinsic rewards directly from latent space geometry. Crucially, our empirical analysis reveals a compelling geometric property: terminal token representations of correct reasoning trajectories form dense clusters with high intra-class similarity, whereas incorrect trajectories remain scattered as outliers. In light of this discovery, we introduce the Iterative Robust Centroid Estimation (IRCE) algorithm, which generates dense, continuous rewards by mitigating magnitude fluctuations via spherical projection and estimating a robust ``truth centroid'' through iterative aggregation. Experimental results on multiple datasets show that our method maintains model performance while achieving a training speedup of over 2x compared to baselines. Furthermore, extensive results demonstrate strong generalization ability and robustness. The code will be released soon.

تم تقديم إطار عمل جديد يسمى Latent-GRPO لتعزيز أداء التفكير لنماذج اللغة الكبيرة (LLMs) من خلال اشتقاق المكافآت الجوهرية من هندسة الفضاء الكامن، مما يعالج قيود تحسين السياسة النسبية الجماعية (GRPO) التقليدي الذي يعتمد على المراجعين الخارجيين.

Se ha introducido un nuevo marco llamado Latent-GRPO para mejorar el rendimiento de razonamiento de los Modelos de Lenguaje de Gran Escala (LLMs) al derivar recompensas intrínsecas de la geometría del espacio latente, abordando las limitaciones de la Optimización Relativa de Grupo (GRPO) tradicional que depende de verificadores externos.

Un nouveau cadre appelé Latent-GRPO a été introduit pour améliorer la performance de raisonnement des grands modèles de langage (LLMs) en dérivant des récompenses intrinsèques de la géométrie de l'espace latent, abordant les limitations de l'optimisation relative de groupe (GRPO) traditionnelle qui dépend des vérificateurs externes.

A new framework called Latent-GRPO has been introduced to enhance the reasoning performance of Large Language Models (LLMs) by deriving intrinsic rewards from latent space geometry, addressing the limitations of traditional Group Relative Policy Optimization (GRPO) that relies on external verifiers.

Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clustering

One More Thing in AI – Your Shortcut to AI Mastery

Group-Aware Reinforcement Learning for Output Diversity in Large Language Models

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

Adaptive Privacy Policy Generator

GPTHumanizer

GPT Rules

CodeSpaced

Ready to build your own newsroom?