arXiv:2510.25803v2 Announce Type: replace 
Abstract: Pre-training has proven effective in addressing data scarcity and performance limitations in solving PDE problems with neural operators. However, challenges remain due to the heterogeneity of PDE datasets in equation types, which leads to high errors in mixed training. Additionally, dense pre-training models that scale parameters by increasing network width or depth incur significant inference costs. To tackle these challenges, we propose a novel Mixture-of-Experts Pre-training Operator Transformer (MoE-POT), a sparse-activated architecture that scales parameters efficiently while controlling inference costs. Specifically, our model adopts a layer-wise router-gating network to dynamically select 4 routed experts from 16 expert networks during inference, enabling the model to focus on equation-specific features. Meanwhile, we also integrate 2 shared experts, aiming to capture common properties of PDE and reduce redundancy among routed experts. The final output is computed as the weighted average of the results from all activated experts. We pre-train models with parameters from 30M to 0.5B on 6 public PDE datasets. Our model with 90M activated parameters achieves up to a 40% reduction in zero-shot error compared with existing models with 120M activated parameters. Additionally, we conduct interpretability analysis, showing that dataset types can be inferred from router-gating network decisions, which validates the rationality and effectiveness of the MoE architecture.

تقدم دراسة جديدة مشغل محول مختلط من الخبراء يهدف إلى تحسين ما قبل التدريب لمشغلات الشبكات العصبية لحل المعادلات التفاضلية الجزئية (PDE). تتناول هذه الطريقة التحديات التي تطرحها تنوع مجموعات بيانات PDE والتكاليف العالية المرتبطة بالنماذج الكثيفة. من خلال تحسين طرق ما قبل التدريب، يمكن أن تعزز هذه الابتكارات الأداء بشكل كبير في تطبيقات متنوعة، مما يجعلها تقدمًا ملحوظًا في مجال التعلم الآلي.

Un nuevo estudio presenta un transformador de mezcla de expertos destinado a mejorar el preentrenamiento de operadores neuronales para resolver ecuaciones en derivadas parciales (EDP). Este enfoque aborda los desafíos que plantea la diversidad de conjuntos de datos de EDP y los altos costos asociados con los modelos densos. Al mejorar los métodos de preentrenamiento, esta innovación podría aumentar significativamente el rendimiento en diversas aplicaciones, lo que la convierte en un avance notable en el campo del aprendizaje automático.

Une nouvelle étude présente un transformateur à mélange d'experts pour améliorer le pré-entraînement des opérateurs neuronaux dans la résolution des équations aux dérivées partielles (EDP). Cette approche répond aux défis posés par la diversité des ensembles de données EDP et aux coûts élevés associés aux modèles denses. En améliorant les méthodes de pré-entraînement, cette innovation pourrait considérablement améliorer les performances dans diverses applications, ce qui en fait une avancée remarquable dans le domaine de l'apprentissage automatique.

A new study introduces a Mixture-of-Experts Operator Transformer aimed at improving the pre-training of neural operators for solving partial differential equations (PDEs). This approach addresses the challenges posed by diverse PDE datasets and the high costs associated with dense models. By enhancing pre-training methods, this innovation could significantly boost performance in various applications, making it a noteworthy advancement in the field of machine learning.

Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training

One More Thing in AI – Your Shortcut to AI Mastery

Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training

Was this article worth reading? Share it

One More Thing in AI

Airparser

LucidQuery AI

Chattermate

Hypertune

The Visualizer

Ready to build your own newsroom?