arXiv:2505.13196v3 Announce Type: replace 
Abstract: We introduce Velocity-Regularized Adam (VRAdam), a physics-inspired optimizer for training deep neural networks that draws on ideas from quartic terms for kinetic energy with its stabilizing effects on various system dynamics. Previous algorithms, including the ubiquitous Adam, operate at the so-called adaptive edge of stability regime during training, leading to rapid oscillations and slowed convergence of loss. However, VRAdam adds a higher order penalty on the learning rate based on the velocity such that the algorithm automatically slows down whenever weight updates become large. In practice, we observe that the effective dynamic learning rate shrinks in high-velocity regimes, and damping oscillations. By combining this velocity-based regularizer for global damping with per-parameter scaling of Adam, we create a powerful hybrid optimizer. For this optimizer, we provide rigorous theoretical analysis of operation at the edge of stability from a physical and control perspective for the momentum. Furthermore, we derive convergence bounds with the rate $\mathcal{O}(\ln(N)/\sqrt{N})$ for a stochastic non convex objective under mild assumptions. We demonstrate that VRAdam exceeds the performance against standard optimizers including AdamW. We benchmark various tasks such as image classification, language modeling, and generative modeling using diverse architectures and training methodologies including Convolutional Neural Networks (CNNs), Transformers, and GFlowNets.

تم تقديم مُحسّن جديد يُدعى Velocity-Regularized Adam (VRAdam) مستوحى من مفاهيم الفيزياء المتعلقة بالطاقة الحركية. يهدف هذا المُحسّن إلى تحسين تدريب الشبكات العصبية العميقة من خلال معالجة الاهتزازات السريعة ومشكلات التقارب البطيء المرتبطة بالخوارزميات الموجودة مثل Adam. من خلال دمج عقوبة من الدرجة الأعلى على معدل التعلم بناءً على السرعة، يعمل VRAdam على استقرار تحديثات الوزن بشكل فعال أثناء التدريب.

Se ha introducido un nuevo optimizador llamado Velocity-Regularized Adam (VRAdam), inspirado en conceptos de física relacionados con la energía cinética. Este optimizador tiene como objetivo mejorar el entrenamiento de redes neuronales profundas al abordar las oscilaciones rápidas y los problemas de convergencia lenta asociados con algoritmos existentes como Adam. Al incorporar una penalización de orden superior sobre la tasa de aprendizaje basada en la velocidad, VRAdam estabiliza eficazmente las actualizaciones de pesos durante el entrenamiento.

Un nouvel optimiseur nommé Velocity-Regularized Adam (VRAdam) a été introduit, inspiré par des concepts de physique liés à l'énergie cinétique. Cet optimiseur vise à améliorer l'entraînement des réseaux de neurones profonds en s'attaquant aux oscillations rapides et aux problèmes de convergence lente associés aux algorithmes existants comme Adam. En incorporant une pénalité d'ordre supérieur sur le taux d'apprentissage basée sur la vitesse, VRAdam stabilise efficacement les mises à jour de poids pendant l'entraînement.

A new optimizer named Velocity-Regularized Adam (VRAdam) has been introduced, inspired by physics concepts related to kinetic energy. This optimizer aims to enhance the training of deep neural networks by addressing the rapid oscillations and slow convergence issues associated with existing algorithms like Adam. By incorporating a higher-order penalty on the learning rate based on velocity, VRAdam effectively stabilizes weight updates during training.