arXiv:2605.29714v1 Announce Type: new 
Abstract: Mixture-of-Experts (MoE) models are widely used to scale language models, yet their expert routing behavior and adaptation in a multilingual setting remain underexplored. In this work, we study multilingual routing dynamics during continual pre-training of an English-centric MoE model on a multilingual corpus, analyzing how expert usage varies across languages. We find that continual multilingual pre-training leads to diffused, language-agnostic routing in early and middle layers, with language specialization primarily emerging in the final layers. We also show that token-level vocabulary overlap between languages plays an important role in how languages are routed. Motivated by these findings, we propose a parameter-efficient adaptation strategy that updates language-specific and shared experts in the final MoE layers. Experiments on MultiBLiMP and Belebele show that our method achieves a strong performance-efficiency trade-off, attaining competitive performance relative to fine-tuning complete final layers, while updating less than 2% of the parameters. Overall, our findings provide insights into where and how language specialization emerges in MoEs during continual pre-training and provide practical insights for low-resource multilingual adaptation. Our code is available at https://github.com/aditi184/moe-routing-adaptation.

تركزت الأبحاث الأخيرة على ديناميات توجيه الخبراء في نماذج Mixture-of-Experts (MoE)، خاصة في السياقات متعددة اللغات. تكشف الدراسة أن التدريب المستمر لنموذج MoE الذي يركز على الإنجليزية على مجموعة بيانات متعددة اللغات يؤدي إلى توجيه غير مرتبط باللغة في الطبقات الأولى، مع ظهور التخصص في الطبقات النهائية. وهذا يبرز أهمية تداخل المفردات على مستوى الرموز في قرارات التوجيه عبر اللغات.

Investigaciones recientes se han centrado en la dinámica de enrutamiento de expertos en modelos Mixture-of-Experts (MoE), especialmente en contextos multilingües. El estudio revela que el preentrenamiento continuo de un modelo MoE centrado en inglés sobre un corpus multilingüe conduce a un enrutamiento agnóstico al idioma en las capas iniciales, con especialización que emerge en las capas finales. Esto resalta la importancia de la superposición de vocabulario a nivel de token en las decisiones de enrutamiento entre idiomas.

Des recherches récentes se sont concentrées sur la dynamique de routage des experts dans les modèles Mixture-of-Experts (MoE), en particulier dans des contextes multilingues. L'étude révèle que le pré-entraînement continu d'un modèle MoE centré sur l'anglais sur un corpus multilingue conduit à un routage agnostique à la langue dans les couches initiales, avec une spécialisation émergeant dans les couches finales. Cela souligne l'importance du chevauchement du vocabulaire au niveau des tokens dans les décisions de routage entre les langues.

Recent research has focused on the dynamics of expert routing in Mixture-of-Experts (MoE) models, particularly in multilingual contexts. The study reveals that continual pre-training of an English-centric MoE model on a multilingual corpus leads to language-agnostic routing in early layers, with specialization emerging in the final layers. This highlights the importance of token-level vocabulary overlap in routing decisions across languages.