arXiv:2509.18542v2 Announce Type: replace 
Abstract: Mixture-of-Experts (MoE) models enable scalable performance by activating large parameter sets sparsely, minimizing computational overhead. To mitigate the prohibitive cost of training MoEs from scratch, recent work employs upcycling, reusing a single pre-trained dense model by replicating its feed-forward network (FFN) layers into experts. However, this limits expert diversity, as all experts originate from a single pre-trained dense model. This paper addresses this limitation by constructing powerful MoE models using experts sourced from multiple identically-architected but disparate pre-trained models (e.g., Qwen2.5-Coder and Qwen2). A key challenge lies in the fact that these source models occupy disparate, dissonant regions of the parameter space, making direct upcycling prone to severe performance degradation. To overcome this, we propose Symphony-MoE, a novel two-stage framework designed to harmonize these models into a single, coherent expert mixture. First, we establish this harmony in a training-free manner: we construct a shared backbone via a layer-aware fusion strategy and, crucially, alleviate parameter misalignment among experts using activation-based functional alignment. Subsequently, a stage of post-training coordinates the entire architecture. Experiments demonstrate that our method successfully integrates experts from heterogeneous sources, achieving an MoE model that significantly surpasses baselines in multi-domain tasks and out-of-distribution generalization.

يقدم إطار Symphony-MoE نهجًا مبتكرًا لتحسين نماذج Mixture-of-Experts (MoE) من خلال دمج الخبراء من نماذج مدربة مسبقًا متعددة. تعالج هذه الطريقة القيود التي كانت تعاني منها النماذج السابقة التي اعتمدت على مصدر واحد، مما قيد تنوع الخبراء. من خلال تنسيق النماذج المتباينة، يحسن Symphony-MoE الأداء بشكل كبير، مما يوضح إمكانياته في تعزيز قدرات الذكاء الاصطناعي.

El marco Symphony-MoE introduce un enfoque innovador para mejorar los modelos Mixture-of-Experts (MoE) al integrar expertos de múltiples modelos preentrenados. Este método aborda las limitaciones de modelos anteriores que dependían de una sola fuente, lo que restringía la diversidad de expertos. Al armonizar modelos dispares, Symphony-MoE mejora significativamente el rendimiento, demostrando su potencial para avanzar en las capacidades de la IA.

Le cadre Symphony-MoE introduit une approche novatrice pour améliorer les modèles Mixture-of-Experts (MoE) en intégrant des experts provenant de plusieurs modèles pré-entraînés. Cette méthode répond aux limitations des modèles précédents qui s'appuyaient sur une seule source, ce qui restreignait la diversité des experts. En harmonisant des modèles disparates, Symphony-MoE améliore considérablement les performances, démontrant son potentiel à faire progresser les capacités de l'IA.

The Symphony-MoE framework introduces a novel approach to enhance Mixture-of-Experts (MoE) models by integrating experts from multiple pre-trained models. This method addresses the limitations of previous models that relied on a single source, which restricted expert diversity. By harmonizing disparate models, Symphony-MoE significantly improves performance, demonstrating its potential to advance AI capabilities.

Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts

Was this article worth reading? Share it

Ready to build your own newsroom?