arXiv:2512.18373v1 Announce Type: cross 
Abstract: Neural network optimization remains one of the most consequential yet poorly understood challenges in modern AI research, where improvements in training algorithms can lead to enhanced feature learning in foundation models, order-of-magnitude reductions in training time, and improved interpretability into how networks learn. While stochastic gradient descent (SGD) and its variants have become the de facto standard for training deep networks, their success in these over-parameterized regimes often appears more empirical than principled. This thesis investigates this apparent paradox by tracing the evolution of optimization algorithms from classical first-order methods to modern higher-order techniques, revealing how principled algorithmic design can demystify the training process. Starting from first principles with SGD and adaptive gradient methods, the analysis progressively uncovers the limitations of these conventional approaches when confronted with anisotropy that is representative of real-world data. These breakdowns motivate the exploration of sophisticated alternatives rooted in curvature information: second-order approximation techniques, layer-wise preconditioning, adaptive learning rates, and more. Next, the interplay between these optimization algorithms and the broader neural network training toolkit, which includes prior and recent developments such as maximal update parametrization, learning rate schedules, and exponential moving averages, emerges as equally essential to empirical success. To bridge the gap between theoretical understanding and practical deployment, this paper offers practical prescriptions and implementation strategies for integrating these methods into modern deep learning workflows.

أطروحة جديدة بعنوان 'Towards Guided Descent' تستكشف خوارزميات التحسين لتدريب الشبكات العصبية، مع التركيز على التطور من الأساليب التقليدية من الدرجة الأولى إلى التقنيات الحديثة من الدرجة الأعلى. تسلط الضوء على قيود الانحدار العشوائي (SGD) ونسخه في الأنظمة المفرطة المعلمات، مما يبرز الحاجة إلى تصميم خوارزمي مبني على مبادئ لتحسين كفاءة التدريب وقابلية التفسير.

Una nueva tesis titulada 'Towards Guided Descent' explora los algoritmos de optimización para el entrenamiento de redes neuronales, centrándose en la evolución de los métodos clásicos de primer orden a técnicas modernas de orden superior. Destaca las limitaciones del descenso de gradiente estocástico (SGD) y sus variantes en regímenes sobreparametrizados, enfatizando la necesidad de un diseño algorítmico fundamentado para mejorar la eficiencia del entrenamiento y la interpretabilidad.

Une nouvelle thèse intitulée 'Towards Guided Descent' explore les algorithmes d'optimisation pour l'entraînement des réseaux de neurones, en se concentrant sur l'évolution des méthodes classiques de premier ordre vers des techniques modernes de plus haut ordre. Elle met en évidence les limites de la descente de gradient stochastique (SGD) traditionnelle et de ses variantes dans des régimes sur-paramétrés, soulignant la nécessité d'une conception algorithmique fondée sur des principes pour améliorer l'efficacité de l'entraînement et l'interprétabilité.

A new thesis titled 'Towards Guided Descent' explores optimization algorithms for training neural networks, focusing on the evolution from classical first-order methods to modern higher-order techniques. It highlights the limitations of traditional stochastic gradient descent (SGD) and its variants in over-parameterized regimes, emphasizing the need for principled algorithmic design to enhance training efficiency and interpretability.