Zeroth-Order Optimization Finds Flat Minima

The study on zeroth-order optimization, published on arXiv, highlights the method's effectiveness in finding flat minima, which are crucial in machine learning applications where gradients are often unavailable. Traditional optimization theories have primarily focused on convergence to arbitrary stationary points, leaving a gap in understanding the implicit regularization that influences the solutions reached. This research fills that gap by demonstrating that zeroth-order optimization favors solutions with a small trace of Hessian, a key factor in distinguishing between sharp and flat minima. The authors provide theoretical convergence rates for approximating flat minima in convex and smooth functions, supported by experimental results from binary classification tasks and language model fine-tuning. These findings not only advance the theoretical framework of optimization but also have practical implications for enhancing the performance of machine learning models in various applicati…