arXiv:2605.27678v1 Announce Type: new 
Abstract: Foundation model training is becoming multimodal, from post-training pipelines to large-scale pretraining. As modality coverage broadens, context windows grow, and encoder LLM scales diverge, a single LLM-centric TP/CP/PP/DP/EP layout increasingly limits throughput. This coupling forces encoders to inherit LLM-driven sharding and placement choices that can add communication, limit encoder parallelism, or constrain the LLM schedule; the mismatch is most pronounced at long contexts, where LLM context parallelism is needed for the fused multimodal sequence but encoder inputs remain bounded. We present heterogeneous parallelism for multimodal large language model training, an abstraction that lets modules in one end-to-end graph use independent layouts and rank placements, supporting colocated execution on shared GPUs and non-colocated execution on disjoint rank sets. The key challenge is preserving boundary tensor semantics across independent layouts: forward activations must be materialized for the destination layout, while backward gradients must be routed back to the source layout. We address this with boundary communicators that implement forward and backward layout transforms, plus scheduling extensions for both placement modes. We evaluate optimized homogeneous, colocated heterogeneous, and non-colocated heterogeneous configurations across multimodal workloads and GPU scales to characterize when added layout and placement freedom exposes a better operating point. Across this sweep, colocated heterogeneity improves TFLOPS/GPU by up to 49.3%, while non-colocated heterogeneity improves aggregate token throughput by up to 13.0% and TFLOPS/GPU by up to 9.6%. We validate loss convergence parity against homogeneous baselines and release the system as an open-source Megatron-LM extension.

تم تقديم نهج جديد لتدريب نماذج اللغة متعددة الوسائط (LLMs) يركز على التوازي غير المتجانس. تتيح هذه الطريقة لمكونات مختلفة داخل نفس الرسم البياني للتدريب استخدام تخطيطات مستقلة وتوزيعات رتب، مما يعزز من خلال ذلك الإنتاجية وكفاءة التنفيذ على وحدات معالجة الرسوميات المشتركة.

Se ha introducido un nuevo enfoque para el entrenamiento de modelos de lenguaje multimodal (LLMs) que se centra en el paralelismo heterogéneo. Este método permite que diferentes módulos dentro de un mismo gráfico de entrenamiento utilicen disposiciones y ubicaciones de rango independientes, mejorando así el rendimiento y la eficiencia de ejecución en GPUs compartidos.

Une nouvelle approche pour l'entraînement des grands modèles de langage multimodaux (LLMs) a été introduite, axée sur le parallélisme hétérogène. Cette méthode permet à différents modules au sein d'un même graphe d'entraînement d'utiliser des agencements et des placements de rang indépendants, améliorant ainsi le débit et l'efficacité d'exécution sur des GPU partagés.

A new approach to training multimodal large language models (LLMs) has been introduced, focusing on heterogeneous parallelism. This method allows different modules within a single training graph to utilize independent layouts and rank placements, enhancing throughput and execution efficiency on shared GPUs.