BSFA: Leveraging the Subspace Dichotomy to Accelerate Neural Network Training

A recent study by BSFA reveals a crucial insight into deep learning optimization, showing that while updates in the dominant eigendirections of the loss Hessian are significant in magnitude, they contribute little to actual loss reduction. Instead, smaller updates in the orthogonal component are driving most of the learning progress. This finding is important as it could lead to more efficient training methods for neural networks, ultimately enhancing their performance and application in various fields.