ADAM Optimization with Adaptive Batch Selection

A recent study challenges the conventional approach to decoupled weight decay in optimization algorithms, specifically questioning the long-held assumption that it should be proportional to the learning rate. The research suggests that a proportionality to the square of the learning rate may be more appropriate, based on steady-state orthogonality arguments. However, findings indicate minimal impact on training dynamics when the perpendicular component of updates is removed.

The paper introduces Arc Gradient Descent (ArcGD), a new optimizer that reformulates traditional gradient descent methods to incorporate phase-aware and user-controlled step dynamics. The evaluation of ArcGD shows it outperforming the Adam optimizer on a non-convex benchmark and a real-world ML dataset, particularly in challenging scenarios like the Rosenbrock function and CIFAR-10 image classification.

The recent study on stochastic approximation with block-coordinate optimal stepsizes introduces adaptive stepsize rules designed to minimize the expected distance from an unknown target point. These rules utilize online estimates of the second moment of the search direction, leading to a new method that competes effectively with the widely used Adam algorithm while requiring less memory and fewer hyper-parameters.

The introduction of the Folded Optimizer with Approximate Moment (FOAM) presents a new approach to training large language models (LLMs) by compressing optimizer states through block-wise gradient means and a residual correction mechanism. This method aims to alleviate memory bottlenecks associated with traditional optimizers like Adam, which are often memory-intensive during training.