arXiv:2605.23093v1 Announce Type: new 
Abstract: Topic modeling in applied psychology increasingly spans two methodological traditions: probabilistic bag-of-words models and newer embedding-based approaches. Yet many evaluations of these methods rely on longer and cleaner benchmark corpora, leaving less guidance for short, open-ended survey responses. This paper compares Structural Topic Models (STM), a probabilistic topic model, and BERTopic, an embedding-based model, for analyzing open-ended survey responses. We evaluated three STM conditions and five BERTopic conditions, varying typographical correction, stemming, embedding choice, and contextual augmentation, a strategy we introduced to provide additional semantic context for very short responses. Results indicate that BERTopic consistently produced higher topic coherence than STM, with contextual augmentation yielding the strongest performance gains. In contrast, higher-dimensional embeddings alone did not improve coherence and were associated with greater data loss. Qualitative evaluation showed that BERTopic generated more interpretable and stable topics, while STM topics were often broader and more mixed. However, STM provides stronger support for inferential covariate analysis, whereas BERTopic covariate comparisons are primarily descriptive. These findings suggest that STM and BERTopic offer complementary strengths. We conclude with practical guidance for selecting and combining topic modeling approaches in applied social science research.

أظهرت دراسة حديثة مقارنة بين نماذج الموضوعات الهيكلية (STM) وBERTopic لتحليل ردود الاستطلاعات المفتوحة القصيرة في علم النفس التطبيقي. شملت التقييمات ظروفًا متغيرة مثل تصحيح الأخطاء الطباعية وزيادة السياق، مما كشف أن BERTopic تفوقت باستمرار على STM في تماسك الموضوعات، خاصة مع إضافة سياق دلالي.

Un estudio reciente comparó los Modelos de Temas Estructurales (STM) y BERTopic para analizar respuestas de encuestas abiertas y breves en psicología aplicada. La evaluación involucró condiciones variables como la corrección tipográfica y la augmentación contextual, revelando que BERTopic superó consistentemente a STM en coherencia temática, especialmente con un contexto semántico adicional.

Une étude récente a comparé les modèles de sujets structurels (STM) et BERTopic pour analyser des réponses d'enquête ouvertes et courtes en psychologie appliquée. L'évaluation a impliqué des conditions variées telles que la correction typographique et l'augmentation contextuelle, révélant que BERTopic surpassait systématiquement STM en cohérence thématique, en particulier avec un contexte sémantique supplémentaire.

A recent study compared Structural Topic Models (STM) and BERTopic for analyzing short, open-ended survey responses in applied psychology. The evaluation involved varying conditions such as typographical correction and contextual augmentation, revealing that BERTopic consistently outperformed STM in topic coherence, particularly with added semantic context.

A Comparative Evaluation of Structural Topic Models and BERTopic for Short, Open-Ended Survey Responses

More coverage on this story

Semantic-Driven Topic Modeling for Analyzing Creativity in Virtual Brainstorming