arXiv:2605.28832v1 Announce Type: new 
Abstract: Topic modeling is a branch of Natural Language Processing (NLP) that aims to organize large collections of texts into coherent groups according to word co-occurrence patterns, with Latent Dirichlet Allocation (LDA) remaining one of the most widely used and interpretable probabilistic approaches. Recent advances in NLP, particularly transformer-based language models, offer improved document representations. It is also known that the size of the model (in terms of number of parameters) has a significant impact in the performance of the language models on different pre-defined tasks. In this study, we systematically examine the effect of model size on topic quality by analyzing the performances of seven transformer-based language models (from small models such as MiniLM to large ones such as LLaMA-2) in a BERTopic pipeline on a variety of corpora. Topic quality is evaluated using coherence and divergence metrics following R{\"o}der et al. (2015). Our results indicate that model size, ranging from 22 million to 13 billion parameters, has a negligible impact on the quality of the topic, suggesting that smaller models can achieve comparable performance to larger models.

أظهرت دراسة حديثة تأثير حجم النموذج على جودة الموضوعات في معالجة اللغة الطبيعية (NLP)، حيث تم التركيز على نماذج اللغة المعتمدة على المحولات مثل MiniLM وLLaMA-2 ضمن خط أنابيب BERTopic. تقيم الدراسة تماسك الموضوعات ومقاييس التباين، مما يبرز أهمية معلمات النموذج في تحسين تمثيلات الوثائق.

Un estudio reciente ha examinado sistemáticamente el impacto del tamaño del modelo en la calidad de los temas en el procesamiento del lenguaje natural (NLP), centrándose en modelos de lenguaje basados en transformadores como MiniLM y LLaMA-2 dentro de un pipeline de BERTopic. La investigación evalúa la coherencia de los temas y las métricas de divergencia, destacando la importancia de los parámetros del modelo para mejorar las representaciones documentales.

Une étude récente a examiné systématiquement l'impact de la taille du modèle sur la qualité des sujets en traitement du langage naturel (NLP), en se concentrant sur des modèles de langage basés sur des transformateurs tels que MiniLM et LLaMA-2 au sein d'un pipeline BERTopic. La recherche évalue la cohérence et les métriques de divergence des sujets, soulignant l'importance des paramètres du modèle pour améliorer les représentations documentaires.

A recent study has systematically examined the impact of model size on topic quality in Natural Language Processing (NLP), focusing on transformer-based language models such as MiniLM and LLaMA-2 within a BERTopic pipeline. The research evaluates topic coherence and divergence metrics, highlighting the significance of model parameters in enhancing document representations.

A comparative study of transformer-based embeddings for topic coherence

arXiv:2606.11196v1 Announce Type: new 
Abstract: Decentralized LLM inference networks need lightweight, reference-free quality evaluation for Proof of Quality (PoQ). We present PoQ-Judge, a framework that trains dedicated judge models to score query-output pairs without ground-truth references. We study three architectures across the quality-cost tradeoff: a TextCNN judge, a MiniLM cross-encoder, and a DeBERTa judge. Using two-stage training on UltraFeedback plus GPT-labeled in-domain data, the best model reaches 0.747 Pearson correlation with the ground-truth proxy on a held-out test set, outperforming reference-based evaluators from prior work. As a reference-free component in composite scoring, it achieves 0.645 Pearson correlation, matching the best single reference-based evaluator while removing the need for reference answers. We also show that online calibration identifies semantic quality as the dominant dimension and that cascade evaluation reduces cost by 72.7 percent with only modest quality loss. Results are much stronger on QA than summarization, pointing to proxy quality as the main remaining limitation.

تم تقديم إطار عمل PoQ-Judge لتوفير تقييم جودة خفيف الوزن وغير مرجعي لشبكات استدلال نماذج اللغة الكبيرة (LLM) اللامركزية. يستخدم نماذج قضاة مخصصة لتقييم أزواج الاستعلام-الإخراج دون الحاجة إلى مراجع حقيقية، مما يظهر تحسينات كبيرة في الأداء مقارنة بالمقيمين المعتمدين على المراجع السابقة.

Se ha presentado el marco PoQ-Judge para proporcionar una evaluación de calidad ligera y sin referencia para las redes de inferencia de grandes modelos de lenguaje (LLM) descentralizados. Utiliza modelos de jueces dedicados para puntuar pares de consulta-salida sin necesidad de referencias de verdad, demostrando mejoras significativas en el rendimiento en comparación con los evaluadores basados en referencias anteriores.

Le cadre PoQ-Judge a été introduit pour fournir une évaluation de qualité légère et sans référence pour les réseaux d'inférence de grands modèles de langage (LLM) décentralisés. Il utilise des modèles de juges dédiés pour évaluer les paires requête-sortie sans nécessiter de références de vérité, montrant des améliorations de performance significatives par rapport aux évaluateurs basés sur des références précédentes.

The PoQ-Judge framework has been introduced to provide a lightweight, reference-free quality evaluation for decentralized large language model (LLM) inference networks. It employs dedicated judge models to score query-output pairs without needing ground-truth references, demonstrating significant performance improvements over previous reference-based evaluators.

A comparative study of transformer-based embeddings for topic coherence

Was this article worth reading? Share it

Ready to build your own newsroom?