Schedulers for Schedule-free: Theoretically inspired hyperparameters

The recent publication on the schedule-free method marks a significant advancement in hyperparameter tuning for deep neural networks. Traditionally, the theory supporting schedule-free methods was limited to a constant learning rate, but this new study extends the last-iterate convergence theory to accommodate any scheduler. This flexibility is crucial as it aligns theoretical insights with practical implementations, particularly those utilizing warm-up schedules. The research demonstrates that the proposed warmup-stable-decay schedule achieves an optimal convergence rate of O(1/sqrt(T)), which is a notable improvement. Additionally, the introduction of a new adaptive Polyak learning rate schedule further enhances performance, proving effective in comparison to various baselines on a black-box model distillation task. These developments not only validate the predictive power of the new convergence theory but also highlight the potential for more efficient machine learning practices, es…