arXiv:2606.09843v3 Announce Type: replace-cross 
Abstract: Large language models (LLMs) give stable answers to personality questionnaires, yet these self-reports fail to predict how the models behave. Is this gap an artifact of forcing human trait categories onto LLMs, or something deeper about LLM self-report? To find out, we built the first psychometric instrument whose dimensions are derived from LLM behavior rather than human psychology. Administering 300 items (240 Likert + 60 scenario) to 25 LLMs across 17 model families, 30 times each, exploratory factor analysis revealed five reliable, replicable factors: Responsiveness, Deference, Boldness, Guardedness, and Verbosity (all Tucker $\phi \geq .957$, all $\alpha \geq .930$). We collected 2,500 open-ended samples and had them rated by 151 humans and a three-judge LLM ensemble. Humans and judges agreed ($\bar{r} = .51$), but self-report predicted neither the ratings nor objective text measures computed from them: the gap persists even for constructs native to LLMs, where a human-mismatch explanation no longer applies. The exception is Verbosity, whose self-report reaches 74% of the criterion-reliability ceiling against human ratings, but does not track raw output length. On Responsiveness, self-report tracked LLM judges ($r = .53$) but not humans ($r = .04$), even though humans and judges otherwise agreed ($r = .59$). This pattern formally rejects any single latent construct driving all three measurements ($p = .007$). Self-report items and LLM judges share a source of variance that human observers do not, and controlling for measurable surface features (length, formatting, enthusiasm markers) does not remove it. This confound is invisible to the within-ensemble reliability checks used to validate LLM judges, and it poses a concrete risk for the LLM-as-judge pipelines now central to model evaluation. We release the instrument as a diagnostic probe for alignment-shaped self-description.

أظهرت دراسة حديثة تطوير أداة نفسية جديدة مصممة لنماذج اللغة الكبيرة (LLMs)، كاشفة عن فجوة كبيرة بين الصفات المبلغ عنها ذاتيًا والسلوك الفعلي عبر 25 نموذجًا. تضمنت الدراسة إجراء 300 عنصر على 25 LLM، مما كشف عن خمسة عوامل متسقة: الاستجابة، التقدير، الجرأة، الحذر، والثرثرة. على الرغم من موثوقية التقارير الذاتية العالية، إلا أنها لم تتوافق مع تقييمات البشر أو المقاييس الموضوعية، مما يدل على استمرار الفجوة بين التقرير الذاتي والسلوك.

Un estudio reciente ha desarrollado un nuevo instrumento psicométrico adaptado para modelos de lenguaje de gran tamaño (LLMs), revelando una brecha significativa entre los rasgos auto-reportados y el comportamiento real en 25 modelos. La investigación consistió en administrar 300 ítems a 25 LLMs, descubriendo cinco factores consistentes: Responsividad, Deferencia, Audacia, Cautela y Verbosidad. A pesar de la alta fiabilidad en los auto-informes, estos no se correlacionaron con las calificaciones humanas ni con las medidas objetivas, indicando una brecha persistente entre auto-informe y comport…

Une étude récente a développé un nouvel instrument psychométrique adapté aux modèles de langage de grande taille (LLMs), révélant un écart significatif entre les traits auto-déclarés et le comportement réel de 25 modèles. La recherche a consisté à administrer 300 éléments à 25 LLMs, découvrant cinq facteurs cohérents : Réactivité, Déférence, Audace, Prudence et Verbosité. Malgré une grande fiabilité des auto-évaluations, celles-ci n'ont pas corrélé avec les évaluations humaines ou les mesures objectives, indiquant un écart persistant entre auto-évaluation et comportement.

A recent study has developed a novel psychometric instrument tailored for large language models (LLMs), revealing a significant gap between self-reported traits and actual behavior across 25 models. The research involved administering 300 items to 25 LLMs, uncovering five consistent factors: Responsiveness, Deference, Boldness, Guardedness, and Verbosity. Despite high reliability in self-reports, these did not correlate with human ratings or objective measures, indicating a persistent self-report-behavior gap.