Non-Euclidean Gradient Descent Operates at the Edge of Stability

A recent study published on arXiv discusses the phenomenon known as the Edge of Stability (EoS) in gradient descent, where the sharpness of the Hessian approaches a stability threshold during optimization processes. This research provides a new interpretation of EoS through Directional Smoothness, extending its applicability to non-Euclidean norms and introducing a generalized sharpness measure that encompasses various gradient descent methods.

The implications of this study are significant for the field of deep learning, as it enhances the understanding of optimization techniques and may lead to improved performance in neural network training by addressing theoretical gaps in existing methods.