arXiv:2512.20638v2 Announce Type: replace-cross 
Abstract: The evaluation of large language models relies heavily on standardized benchmarks. These benchmarks provide useful aggregated metrics, but can obscure (i) particular sub-areas where the models are weak ("model gaps") and (ii) imbalanced coverage in the benchmarks themselves ("benchmark gaps"). To automatically uncover both types of gaps, we propose a simple new method using concept activations from sparse autoencoders, to identify fine-grained gaps on a per-concept basis. The method also benefits from grounding evaluation in the model's internal representations, as well as easy comparison across benchmarks. We applied the method to five popular open-source models and more than a dozen benchmarks, as illustrative examples. As validation of the approach, we found that our automatic, unsupervised method was able to recover model gaps that have been previously documented in the literature (e.g. relating to sycophancy), in addition to identifying novel model gaps. We were also able to automatically uncover benchmark gaps: core concepts that should fall within the scope of a given benchmark. Our "competency gaps" method can be used to complement existing benchmarks, by providing a concept-level decomposition of model behavior, and by helping benchmark developers iterate upon benchmark design. Code is available at https://competency-gaps.github.io.

أظهرت دراسة حديثة وجود فجوات كبيرة في كفاءة نماذج اللغة الكبيرة (LLMs) ومعاييرها، مما يبرز نقاط الضعف في مجالات معينة والتغطية غير المتوازنة في المعايير الحالية. تقدم الدراسة طريقة جديدة تستخدم تنشيطات المفاهيم من أجهزة التشفير النادرة لتحديد هذه الفجوات على مستوى دقيق، حيث تم تطبيقها على عدة نماذج مفتوحة المصدر ومعايير.

Un estudio reciente ha revelado importantes brechas de competencia en los modelos de lenguaje de gran tamaño (LLMs) y sus benchmarks, destacando debilidades en áreas específicas y una cobertura desequilibrada en los benchmarks existentes. La investigación introduce un nuevo método que utiliza activaciones conceptuales de autoencoders dispersos para identificar estas brechas a un nivel granular, aplicándolo a varios modelos de código abierto y benchmarks.

Une étude récente a révélé des lacunes significatives dans les compétences des modèles de langage de grande taille (LLMs) et leurs benchmarks, mettant en évidence des faiblesses dans des sous-domaines spécifiques et une couverture déséquilibrée dans les benchmarks existants. La recherche introduit une nouvelle méthode utilisant des activations conceptuelles provenant d'autoencodeurs épars pour identifier ces lacunes de manière granulaire, en l'appliquant à divers modèles open-source et benchmarks.

A recent study has revealed significant competency gaps in large language models (LLMs) and their benchmarks, highlighting weaknesses in specific sub-areas and imbalanced coverage in existing benchmarks. The research introduces a novel method utilizing concept activations from sparse autoencoders to identify these gaps on a granular level, applying it to various open-source models and benchmarks.