arXiv:2510.04988v3 Announce Type: replace 
Abstract: The vast majority of modern deep learning models are trained with momentum-based first-order optimizers. The momentum term governs the optimizer's memory by determining how much each past gradient contributes to the current convergence direction. Fundamental momentum methods, such as Nesterov Accelerated Gradient and the Heavy Ball method, as well as more recent optimizers such as AdamW and Lion, all rely on the momentum coefficient that is customarily set to $\beta = 0.9$ and kept constant during model training, a strategy widely used by practitioners, yet suboptimal. In this paper, we introduce an \textit{adaptive memory} mechanism that replaces constant momentum with a dynamic momentum coefficient that is adjusted online during optimization. We derive our method by approximating the objective function using two planes: one derived from the gradient at the current iterate and the other obtained from the accumulated memory of the past gradients. To the best of our knowledge, such a proximal framework was never used for momentum-based optimization. Our proposed approach is novel, extremely simple to use, and does not rely on extra assumptions or hyperparameter tuning. We implement adaptive memory variants of both SGD and AdamW across a wide range of learning tasks, from simple convex problems to large-scale deep learning scenarios, demonstrating that our approach can outperform standard SGD and Adam with hand-tuned momentum coefficients. Finally, our work opens doors for new ways of inducing adaptivity in optimization.

يقدم مقال جديد آلية ذاكرة تكيفية لتحسين التعلم العميق، حيث يتم استبدال معامل الزخم الثابت التقليدي بمعامل ديناميكي يتكيف أثناء التدريب. تهدف هذه الطريقة إلى تحسين تقارب النماذج المدربة باستخدام المحسنات من الدرجة الأولى المعتمدة على الزخم، مثل AdamW وطريقة تسرع نستر.

Un nuevo artículo presenta un mecanismo de memoria adaptativa para la optimización del aprendizaje profundo, reemplazando el coeficiente de momento constante tradicional por uno dinámico que se ajusta durante el entrenamiento. Este método busca mejorar la convergencia de los modelos entrenados con optimizadores de primer orden basados en momento, como AdamW y el Gradiente Acelerado de Nesterov.

Un nouvel article présente un mécanisme de mémoire adaptative pour l'optimisation de l'apprentissage profond, remplaçant le coefficient de momentum constant traditionnel par un dynamique qui s'ajuste pendant l'entraînement. Cette méthode vise à améliorer la convergence des modèles entraînés avec des optimisateurs de premier ordre basés sur le momentum, tels qu'AdamW et le gradient accéléré de Nesterov.

A new paper introduces an adaptive memory mechanism for deep learning optimization, replacing the traditional constant momentum coefficient with a dynamic one that adjusts during training. This method aims to enhance the convergence of models trained with momentum-based first-order optimizers, such as AdamW and Nesterov Accelerated Gradient.