arXiv:2411.00918v2 Announce Type: replace-cross 
Abstract: Mixture of experts (MoE) architectures have become a cornerstone for scaling up and are a key component in most large language models such as GPT-OSS, DeepSeek-V3, Llama-4, and Gemini-2.5. However, systematic research on MoE remains severely constrained by the prohibitive computational costs of training and evaluation, restricting large-scale studies accessible to most researchers. We introduce LibMoE, a unified framework for reproducible, efficient, and extensible MoE research that supports both pretraining and sparse-upcycling regimes. Beyond unified implementations, the framework provides transparent analytical tools for probing routing and expert dynamics. Leveraging this foundation, we conduct a comprehensive analysis along three dimensions: (i) routing dynamics, covering expert selection patterns, routing stability and optimality, and how routing entropy reveals task specialization and expert diversity; (ii) the effect of lightweight initialization on load balancing, demonstrating how subtle changes in router initialization shape early expert utilization; and (iii) training regime differences, revealing how sparse upcycling and full pretraining exhibit distinct routing patterns and stability profiles. By lowering the barrier to entry and standardizing evaluation, along with our comprehensive analysis, LibMoE broadens access to MoE research and establishes a reliable benchmark to guide future innovations. Project page: https://fsoft-aic.github.io/fsoft-LibMoE.github.io.

تقديم LibMoE يمثل تقدمًا كبيرًا في مجال الذكاء الاصطناعي، خاصة في تقييم هياكل Mixture of Experts (MoE) المستخدمة في نماذج اللغة الكبيرة. يهدف هذا الإطار الجديد إلى تخفيف التحديات التي تطرحها التكاليف الحاسوبية العالية، مما يسهل على الباحثين إجراء دراسات واسعة النطاق. من خلال توفير منصة موحدة للبحث القابل للتكرار، يمكن أن يعمل LibMoE على ديمقراطية الوصول إلى تقنيات الذكاء الاصطناعي المتطورة، مما يعزز الابتكار والتعاون في هذا المجال.

La introducción de LibMoE marca un avance significativo en el campo de la inteligencia artificial, especialmente en la evaluación de arquitecturas de Mixture of Experts (MoE) utilizadas en grandes modelos de lenguaje. Este nuevo marco tiene como objetivo aliviar los desafíos que presentan los altos costos computacionales, facilitando a los investigadores la realización de estudios a gran escala. Al proporcionar una plataforma unificada para la investigación reproducible, LibMoE podría democratizar el acceso a tecnologías de IA de vanguardia, fomentando la innovación y la colaboración en el campo.

L'introduction de LibMoE représente une avancée significative dans le domaine de l'intelligence artificielle, en particulier dans l'évaluation des architectures de Mixture of Experts (MoE) utilisées dans les grands modèles de langage. Ce nouveau cadre vise à atténuer les défis posés par les coûts computationnels élevés, facilitant ainsi la réalisation d'études à grande échelle par les chercheurs. En offrant une plateforme unifiée pour la recherche reproductible, LibMoE pourrait démocratiser l'accès aux technologies d'IA de pointe, favorisant l'innovation et la collaboration dans le domaine.

The introduction of LibMoE marks a significant advancement in the field of artificial intelligence, particularly in the benchmarking of Mixture of Experts (MoE) architectures used in large language models. This new framework aims to alleviate the challenges posed by high computational costs, making it easier for researchers to conduct large-scale studies. By providing a unified platform for reproducible research, LibMoE could democratize access to cutting-edge AI technologies, fostering innovation and collaboration in the field.

LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models

arXiv:2511.19333v1 Announce Type: new 
Abstract: Test-time scaling, which leverages additional computation during inference to improve model accuracy, has enabled a new class of Large Language Models (LLMs) that are able to reason through complex problems by understanding the goal, turning this goal into a plan, working through intermediate steps, and checking their own work before answering . Frontier large language models with reasoning capabilities, such as DeepSeek-R1 and OpenAI's gpt-oss, follow the same procedure when solving complex problems by generating intermediate reasoning traces before giving the final answer. Today, these models are being increasingly used to generate reasoning traces that serve as high-quality supervised data for post-training of small and medium-sized language models to teach reasoning capabilities without requiring expensive human curation. In this work, we compare the performance of medium-sized LLMs on Math problems after post-training on two kinds of reasoning traces. We compare the impact of reasoning traces generated by DeepSeek-R1 and gpt-oss LLMs in terms of accuracy and inference efficiency.

أدت التطورات الأخيرة في نماذج اللغة الكبيرة (LLMs) إلى إدخال تقنيات توسيع وقت الاختبار التي تعزز قدرات التفكير، كما يتضح من نماذج DeepSeek-R1 وgpt-oss من OpenAI. تقوم هذه النماذج بتوليد آثار تفكير وسيطة لتحسين الدقة في حل المشكلات المعقدة، مما يسمح بإعادة تدريب فعالة للنماذج الأصغر دون الحاجة إلى تدخل بشري مكثف.

Los avances recientes en modelos de lenguaje de gran tamaño (LLMs) han introducido técnicas de escalado en el momento de la prueba que mejoran las capacidades de razonamiento, como lo demuestran los modelos DeepSeek-R1 y gpt-oss de OpenAI. Estos modelos generan trazas de razonamiento intermedias para mejorar la precisión en la resolución de problemas complejos, lo que permite un post-entrenamiento efectivo de modelos más pequeños sin requerir una intervención humana extensa.

Les avancées récentes dans les modèles de langage de grande taille (LLMs) ont introduit des techniques d'échelle au moment du test qui améliorent les capacités de raisonnement, comme le montrent les modèles DeepSeek-R1 et gpt-oss d'OpenAI. Ces modèles génèrent des traces de raisonnement intermédiaires pour améliorer la précision dans la résolution de problèmes complexes, permettant ainsi un post-entraînement efficace de modèles plus petits sans nécessiter une intervention humaine extensive.

Recent advancements in large language models (LLMs) have introduced test-time scaling techniques that enhance reasoning capabilities, as demonstrated by models like DeepSeek-R1 and OpenAI's gpt-oss. These models generate intermediate reasoning traces to improve accuracy in solving complex problems, allowing for effective post-training of smaller models without extensive human input.

LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models

Was this article worth reading? Share it

Augmeta

Langtail

Sellm