arXiv:2411.00918v2 Announce Type: replace-cross 
Abstract: Mixture of experts (MoE) architectures have become a cornerstone for scaling up and are a key component in most large language models such as GPT-OSS, DeepSeek-V3, Llama-4, and Gemini-2.5. However, systematic research on MoE remains severely constrained by the prohibitive computational costs of training and evaluation, restricting large-scale studies accessible to most researchers. We introduce LibMoE, a unified framework for reproducible, efficient, and extensible MoE research that supports both pretraining and sparse-upcycling regimes. Beyond unified implementations, the framework provides transparent analytical tools for probing routing and expert dynamics. Leveraging this foundation, we conduct a comprehensive analysis along three dimensions: (i) routing dynamics, covering expert selection patterns, routing stability and optimality, and how routing entropy reveals task specialization and expert diversity; (ii) the effect of lightweight initialization on load balancing, demonstrating how subtle changes in router initialization shape early expert utilization; and (iii) training regime differences, revealing how sparse upcycling and full pretraining exhibit distinct routing patterns and stability profiles. By lowering the barrier to entry and standardizing evaluation, along with our comprehensive analysis, LibMoE broadens access to MoE research and establishes a reliable benchmark to guide future innovations. Project page: https://fsoft-aic.github.io/fsoft-LibMoE.github.io.

تقديم LibMoE يمثل تقدمًا كبيرًا في مجال الذكاء الاصطناعي، خاصة في تقييم هياكل Mixture of Experts (MoE) المستخدمة في نماذج اللغة الكبيرة. يهدف هذا الإطار الجديد إلى تخفيف التحديات التي تطرحها التكاليف الحاسوبية العالية، مما يسهل على الباحثين إجراء دراسات واسعة النطاق. من خلال توفير منصة موحدة للبحث القابل للتكرار، يمكن أن يعمل LibMoE على ديمقراطية الوصول إلى تقنيات الذكاء الاصطناعي المتطورة، مما يعزز الابتكار والتعاون في هذا المجال.

La introducción de LibMoE marca un avance significativo en el campo de la inteligencia artificial, especialmente en la evaluación de arquitecturas de Mixture of Experts (MoE) utilizadas en grandes modelos de lenguaje. Este nuevo marco tiene como objetivo aliviar los desafíos que presentan los altos costos computacionales, facilitando a los investigadores la realización de estudios a gran escala. Al proporcionar una plataforma unificada para la investigación reproducible, LibMoE podría democratizar el acceso a tecnologías de IA de vanguardia, fomentando la innovación y la colaboración en el campo.

L'introduction de LibMoE représente une avancée significative dans le domaine de l'intelligence artificielle, en particulier dans l'évaluation des architectures de Mixture of Experts (MoE) utilisées dans les grands modèles de langage. Ce nouveau cadre vise à atténuer les défis posés par les coûts computationnels élevés, facilitant ainsi la réalisation d'études à grande échelle par les chercheurs. En offrant une plateforme unifiée pour la recherche reproductible, LibMoE pourrait démocratiser l'accès aux technologies d'IA de pointe, favorisant l'innovation et la collaboration dans le domaine.

The introduction of LibMoE marks a significant advancement in the field of artificial intelligence, particularly in the benchmarking of Mixture of Experts (MoE) architectures used in large language models. This new framework aims to alleviate the challenges posed by high computational costs, making it easier for researchers to conduct large-scale studies. By providing a unified platform for reproducible research, LibMoE could democratize access to cutting-edge AI technologies, fostering innovation and collaboration in the field.

LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models

arXiv:2508.16151v2 Announce Type: replace-cross 
Abstract: The rapid advancement of Large Language Models (LLMs) has established language as a core general-purpose cognitive substrate, driving the demand for specialized Language Processing Units (LPUs) tailored for LLM inference. To overcome the growing energy consumption of LLM inference systems, this paper proposes a Hardwired-Neurons Language Processing Unit (HNLPU), which physically hardwires LLM weight parameters into the computational fabric, achieving several orders of magnitude computational efficiency improvement by extreme specialization. However, a significant challenge still lies in the scale of modern LLMs. A straightforward hardwiring of gpt-oss 120 B would require fabricating photomask sets valued at over 6 billion dollars, rendering this straightforward solution economically impractical. Addressing this challenge, we propose the novel Metal-Embedding methodology. Instead of embedding weights in a 2D grid of silicon device cells, Metal-Embedding embeds weight parameters into the 3D topology of metal wires. This brings two benefits: (1) a 15x increase in density, and (2) 60 out of 70 photomask layers are homogeneous across chips, including all EUV photomasks. In total, Metal-Embedding reduced the photomask cost by 112x, bringing the Non-Recurring Engineering (NRE) cost of HNLPU into an economically viable range. Experimental results show that HNLPU achieved 249,960 tokens/s (5,555x/85x that of GPU/WSE), 36 tokens/J (1,047x/283x that of GPU/WSE), 13,232 mm2 total die area, $59.46 M-123.5 M estimated NRE at 5 nm technology. Analysis shows that HNLPU achieved 41.7-80.4x improvement in cost-effectiveness and 357x reduction in carbon footprint compared to OpenAI-scale H100 clusters, under an annual weight updating assumption.

يهدف تطوير وحدات معالجة اللغة ذات الخلايا العصبية الموصلة (HNLPU) إلى تعزيز كفاءة نماذج اللغة الكبيرة (LLMs) من خلال توصيل معلمات الوزن فعليًا في الهيكل الحسابي، مما يحقق تحسينًا كبيرًا في الكفاءة الحسابية. ومع ذلك، تواجه الجدوى الاقتصادية لهذا النهج تحديات بسبب التكاليف العالية المرتبطة بتصنيع مجموعات القناع الضوئي لنماذج اللغة الكبيرة الحديثة، مثل gpt-oss 120 B.

El desarrollo de las Unidades de Procesamiento de Lenguaje con Neuronas Cableadas (HNLPU) tiene como objetivo mejorar la eficiencia de los Modelos de Lenguaje Grande (LLM) al cablear físicamente los parámetros de peso en la estructura computacional, logrando una mejora significativa en la eficiencia computacional. Sin embargo, la viabilidad económica de este enfoque se ve desafiada por los altos costos asociados con la fabricación de juegos de fotomáscaras para los LLM modernos, como gpt-oss 120 B.

Le développement des unités de traitement du langage à neurones câblés (HNLPU) vise à améliorer l'efficacité des modèles de langage de grande taille (LLM) en câblant physiquement les paramètres de poids dans le tissu computationnel, ce qui améliore considérablement l'efficacité computationnelle. Cependant, la faisabilité économique de cette approche est remise en question par les coûts élevés associés à la fabrication de jeux de photomasques pour les LLM modernes, tels que gpt-oss 120 B.

The development of Hardwired-Neurons Language Processing Units (HNLPU) aims to enhance the efficiency of Large Language Models (LLMs) by physically hardwiring weight parameters into the computational fabric, significantly improving computational efficiency. However, the economic feasibility of this approach is challenged by the high costs associated with fabricating photomask sets for modern LLMs, such as gpt-oss 120 B.

LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models

Was this article worth reading? Share it

Magicley AI

Airparser

LucidQuery AI

Langfuse

Sellm

ChatOne

Ready to build your own newsroom?