arXiv:2605.09176v1 Announce Type: new 
Abstract: Training large language models requires optimization algorithms that are not only statistically effective, but also computationally and memory efficient at extreme scale. Although Adam remains the dominant optimizer for large-scale language-model pretraining and fine-tuning, recent work has revisited nearly every component of the optimization stack: adaptive moment estimation, decoupled weight decay, memory footprint, curvature approximation, sign-based updates, large-batch stability, low-rank gradient structure, and matrix-wise orthogonalized updates. This survey reviews optimizer design for large language models through a systems-and-optimization lens. We organize the literature into classical first-order optimizers, adaptive optimizers, memory-efficient variants, second-order and curvature-aware methods, sign-based and discovered optimizers, low-rank and projection-based methods, and matrix-based optimizers such as Muon. We also discuss benchmarking methodology, including hyperparameter fairness, scale dependence, wall-clock efficiency, token efficiency, memory overhead, and downstream evaluation. We argue that optimizer research for LLMs is entering a new phase: moving from single-algorithm speedup claims toward rigorous, scale-aware comparisons that jointly evaluate convergence, stability, memory, and implementation complexity.

تم تسليط الضوء على التقدم الأخير في خوارزميات التحسين لنماذج اللغة الكبيرة (LLMs)، مع التركيز بشكل خاص على مُحسّن Muon، الذي يتناول التحديات في المناظر التدريبية عالية الأبعاد. تستعرض هذه الدراسة مجموعة متنوعة من المحسنات، بما في ذلك AdamW، وتستكشف فعاليتها من حيث الكفاءة الحاسوبية والذاكرة.

Se han destacado avances recientes en algoritmos de optimización para grandes modelos de lenguaje (LLMs), centrándose especialmente en el optimizador Muon, que aborda desafíos en paisajes de entrenamiento de alta dimensión. Esta encuesta revisa varios optimizadores, incluido AdamW, y explora su efectividad en términos de eficiencia computacional y de memoria.

Des avancées récentes dans les algorithmes d'optimisation pour les grands modèles de langage (LLMs) ont été mises en avant, en particulier en ce qui concerne l'optimiseur Muon, qui aborde les défis dans les paysages d'entraînement de haute dimension. Cette enquête passe en revue divers optimiseurs, y compris AdamW, et explore leur efficacité en termes d'efficacité computationnelle et de mémoire.

Recent advancements in optimization algorithms for large language models (LLMs) have been highlighted, particularly focusing on the Muon optimizer, which addresses challenges in high-dimensional training landscapes. This survey reviews various optimizers, including AdamW, and explores their effectiveness in terms of computational and memory efficiency.