ROOT: Robust Orthogonalized Optimizer for Neural Network Training

In August, the parents of 16-year-old Adam Raine filed a lawsuit against OpenAI and its CEO, Sam Altman, claiming wrongful death after their son died by suicide. OpenAI has responded by asserting that the teenager misused its chatbot, ChatGPT, which allegedly encouraged him to seek help multiple times prior to his death.

HVAdam, a novel full-dimension adaptive optimizer, has been introduced to address the performance gap between adaptive optimizers like Adam and non-adaptive methods such as SGD, particularly in training large-scale models. The new optimizer features continuously tunable adaptivity and a mechanism called incremental delay update (IDU) to enhance convergence across diverse optimization landscapes.

A novel framework named Li_2 has been proposed to characterize the phenomenon of grokking, which involves delayed generalization in machine learning. This framework outlines three key stages of learning dynamics in 2-layer nonlinear networks: lazy learning, independent feature learning, and interactive feature learning, providing insights into how models learn from complex structured inputs.

A new topology-informed inverse rendering approach has been introduced for reconstructing high-genus surface meshes from multi-view images, addressing the limitations of existing methods that struggle with complex geometries. This method utilizes an adaptive V-cycle remeshing scheme alongside a re-parametrized Adam optimizer to enhance both topological and geometric awareness, significantly improving the quality of mesh representations.

A new study published on arXiv explores the use of floating-point approximations in Newton's method for minimizing loss functions in machine learning. The research highlights the advantages of higher-order optimization techniques, demonstrating that mixed-precision Newton optimizers can achieve better accuracy and faster convergence compared to traditional first-order methods like Adam, particularly on datasets such as Australian and MUSH.

Recent advancements in optimization algorithms for neural networks have led to the development of KL-Shampoo and KL-SOAP, which utilize Kullback-Leibler divergence minimization to enhance performance while reducing memory overhead compared to traditional methods like Shampoo and SOAP. These innovations aim to improve the efficiency of neural network training processes.

The Muon optimizer has been theoretically analyzed, demonstrating strong empirical performance and potential as a successor to standard optimizers like AdamW. The study provides convergence proofs across various settings, examining the effects of Nesterov momentum and weight decay on its performance. Additionally, it identifies the critical batch size that minimizes training costs, highlighting the relationship between hyperparameters and efficiency.