Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
This addresses the challenge of efficient and effective training for machine learning practitioners by providing an incremental improvement in learning rate scheduling based on empirical evidence about minima density.
The paper tackles the problem of improving neural network generalization by investigating the density of wide minima and proposes a novel explore-exploit learning rate schedule, resulting in up to 0.84% higher accuracy or 57% reduced training time on image and natural language datasets, including achieving state-of-the-art accuracy on IWSLT'14 (DE-EN).
Several papers argue that wide minima generalize better than narrow minima. In this paper, through detailed experiments that not only corroborate the generalization properties of wide minima, we also provide empirical evidence for a new hypothesis that the density of wide minima is likely lower than the density of narrow minima. Further, motivated by this hypothesis, we design a novel explore-exploit learning rate schedule. On a variety of image and natural language datasets, compared to their original hand-tuned learning rate baselines, we show that our explore-exploit schedule can result in either up to 0.84% higher absolute accuracy using the original training budget or up to 57% reduced training time while achieving the original reported accuracy. For example, we achieve state-of-the-art (SOTA) accuracy for IWSLT'14 (DE-EN) dataset by just modifying the learning rate schedule of a high performing model.