arXiv:2510.22980v2 Announce Type: replace-cross 
Abstract: The growing adoption of spectrum-aware matrix-valued optimizers such as Muon and Shampoo in deep learning motivates a systematic study of their generalization properties and, in particular, when they might outperform competitive algorithms. We approach this question by introducing appropriate simplifying abstractions as follows: First, we use imbalanced data as a testbed. Second, we study the canonical form of such optimizers, which is Spectral Gradient Descent (SpecGD) -- each update step is $UV^T$ where $U\Sigma V^T$ is the truncated SVD of the gradient. Third, within this framework we identify a canonical setting for which we precisely quantify when SpecGD outperforms vanilla Euclidean GD. For a Gaussian mixture data model and both linear and bilinear models, we show that unlike GD, which prioritizes learning dominant principal components of the data first, SpecGD learns all principal components of the data at equal rates. We demonstrate how this translates to a growing gap in balanced accuracy favoring SpecGD early in training and further show that the gap remains consistent even when the GD counterpart uses adaptive step-sizes via normalization. By extending the analysis to deep linear models, we show that depth amplifies these effects. We empirically verify our theoretical findings on a variety of imbalanced datasets. Our experiments compare practical variants of spectral methods, like Muon and Shampoo, against their Euclidean counterparts and Adam. The results validate our findings that these spectral optimizers achieve superior generalization by promoting a more balanced learning of the data's underlying components.

تسلط دراسة حديثة الضوء على مزايا التصميم الطيفي لموين في تعزيز التعميم، خاصة عند تطبيقه على البيانات غير المتوازنة. تُظهر الأبحاث أن طريقة الانحدار الطيفي (SpecGD) تتفوق على تقنيات الانحدار التقليدية من خلال تعلم جميع المكونات الرئيسية للبيانات بمعدلات متساوية، بدلاً من التركيز أولاً على المكونات السائدة.

Un estudio reciente destaca las ventajas del diseño espectral de Muon en la mejora de la generalización, especialmente cuando se aplica a datos desbalanceados. La investigación demuestra que el método de Descenso de Gradiente Espectral (SpecGD) supera a las técnicas tradicionales de descenso de gradiente al aprender todos los componentes principales de los datos a tasas iguales, en lugar de centrarse primero en los componentes dominantes.

Une étude récente met en lumière les avantages de la conception spectrale de Muon pour améliorer la généralisation, en particulier lorsqu'elle est appliquée à des données déséquilibrées. La recherche démontre que la méthode de descente de gradient spectral (SpecGD) surpasse les techniques de descente de gradient traditionnelles en apprenant tous les composants principaux des données à des taux égaux, plutôt qu'en se concentrant d'abord sur les composants dominants.

A recent study highlights the advantages of Muon's spectral design in enhancing generalization, particularly when applied to imbalanced data. The research demonstrates that the Spectral Gradient Descent (SpecGD) method outperforms traditional gradient descent techniques by learning all principal components of data at equal rates, rather than focusing on dominant components first.

How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data

arXiv:2512.08217v1 Announce Type: new 
Abstract: Decoupled weight decay, solely responsible for the performance advantage of AdamW over Adam, has long been set to proportional to learning rate $\gamma$ without questioning. Some researchers have recently challenged such assumption and argued that decoupled weight decay should be set $\propto \gamma^2$ instead based on orthogonality arguments at steady state. To the contrary, we find that eliminating the contribution of the perpendicular component of the update to the weight norm leads to little change to the training dynamics. Instead, we derive that decoupled weight decay $\propto \gamma^2$ results in stable weight norm based on the simple assumption that updates become independent of the weights at steady state, regardless of the nature of the optimizer. Based on the same assumption, we derive and empirically verify that the Total Update Contribution (TUC) of a minibatch under the Scion optimizer is better characterized by the momentum-dependent effective learning rate whose optimal value transfers and we show that decoupled weight decay $\propto \gamma^2$ leads to stable weight and gradient norms and allows us to better control the training dynamics and improve the model performance.

دراسة حديثة تتحدى النهج التقليدي لوزن الانخفاض المنفصل في خوارزميات التحسين، حيث تتساءل عن الافتراض الذي ظل قائمًا لفترة طويلة بأن الوزن يجب أن يكون متناسبًا مع معدل التعلم. تقترح الأبحاث أن يكون التناسب مع مربع معدل التعلم أكثر ملاءمة، بناءً على حجج الاستقامة في الحالة الثابتة. ومع ذلك، تشير النتائج إلى تأثير ضئيل على ديناميات التدريب عند إزالة المكون العمودي للتحديثات.

Un estudio reciente desafía el enfoque convencional sobre la decadencia de peso desacoplada en los algoritmos de optimización, cuestionando la suposición de que debería ser proporcional a la tasa de aprendizaje. La investigación sugiere que una proporcionalidad al cuadrado de la tasa de aprendizaje podría ser más adecuada, basada en argumentos de ortogonalidad en estado estacionario. Sin embargo, los hallazgos indican un impacto mínimo en la dinámica de entrenamiento cuando se elimina la componente perpendicular de las actualizaciones.

Une étude récente remet en question l'approche conventionnelle de la décadence de poids découplée dans les algorithmes d'optimisation, en interrogeant l'hypothèse longtemps tenue selon laquelle elle devrait être proportionnelle au taux d'apprentissage. La recherche suggère qu'une proportionnalité au carré du taux d'apprentissage pourrait être plus appropriée, basée sur des arguments d'orthogonalité à l'état stationnaire. Cependant, les résultats indiquent un impact minimal sur la dynamique d'entraînement lorsque la composante perpendiculaire des mises à jour est supprimée.

A recent study challenges the conventional approach to decoupled weight decay in optimization algorithms, specifically questioning the long-held assumption that it should be proportional to the learning rate. The research suggests that a proportionality to the square of the learning rate may be more appropriate, based on steady-state orthogonality arguments. However, findings indicate minimal impact on training dynamics when the perpendicular component of updates is removed.

How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data

Was this article worth reading? Share it

Magicley AI

Hypertune

Keywords AI