Dynamics of Learning: Generative Schedules from Latent ODEs
This addresses the challenge of inefficient learning rate schedules for neural network optimization, offering a novel, optimizer-agnostic solution that can enhance training across various models.
The paper tackles the problem of learning rate scheduling by modeling training performance as a dynamical system, using latent ODEs to predict optimal schedules from hyperparameter search data, achieving state-of-the-art results in image classification and next-token prediction with improved generalization.
The learning rate schedule is one of the most impactful aspects of neural network optimization, yet most schedules either follow simple parametric functions or react only to short-term training signals. None of them are supported by a comprehensive temporal view of how well neural networks actually train. We present a new learning rate scheduler that models the training performance of neural networks as a dynamical system. It leverages training runs from a hyperparameter search to learn a latent representation of the training process. Given current training metrics, it predicts the future learning rate schedule with the best long-term validation performance. Our scheduler generalizes beyond previously observed training dynamics and creates specialized schedules that deviate noticeably from common parametric functions. It achieves SOTA results for image classification with CNN and ResNet models as well as for next-token prediction with a transformer model. The trained models are located in flatter regions of the loss landscape and thus provide better generalization than those trained with other schedules. Our method is computationally efficient, optimizer-agnostic, and can easily be layered on top of ML experiment-tracking platforms. An implementation of our scheduler will be made available after acceptance.