Toward Optimal Run Racing: Application to Deep Learning Calibration
This work addresses the algorithm calibration problem for deep learning practitioners, offering an incremental improvement in efficiency for selecting neural architectures and hyperparameters.
The paper tackles the expensive calibration problem for deep neural networks by optimally reducing it through early stopping of non-optimal runs, achieving a principled and consistent improvement on state-of-the-art benchmarks like Cifar10, PTB, and Wiki with no extra hyperparameters.
This paper aims at one-shot learning of deep neural nets, where a highly parallel setting is considered to address the algorithm calibration problem - selecting the best neural architecture and learning hyper-parameter values depending on the dataset at hand. The notoriously expensive calibration problem is optimally reduced by detecting and early stopping non-optimal runs. The theoretical contribution regards the optimality guarantees within the multiple hypothesis testing framework. Experimentations on the Cifar10, PTB and Wiki benchmarks demonstrate the relevance of the approach with a principled and consistent improvement on the state of the art with no extra hyper-parameter.