Model-based Asynchronous Hyperparameter and Neural Architecture Search
This work addresses the challenge of efficiently optimizing hyperparameters and neural architectures for machine learning practitioners, representing an incremental improvement by integrating existing techniques into a novel framework.
The paper tackles the problem of hyperparameter and neural architecture search by introducing a model-based asynchronous multi-fidelity method that combines asynchronous Hyperband and Gaussian process-based Bayesian optimization, resulting in substantial speed-ups over state-of-the-art methods on benchmarks for tabular data, image classification, and language modeling.
We introduce a model-based asynchronous multi-fidelity method for hyperparameter and neural architecture search that combines the strengths of asynchronous Hyperband and Gaussian process-based Bayesian optimization. At the heart of our method is a probabilistic model that can simultaneously reason across hyperparameters and resource levels, and supports decision-making in the presence of pending evaluations. We demonstrate the effectiveness of our method on a wide range of challenging benchmarks, for tabular data, image classification and language modelling, and report substantial speed-ups over current state-of-the-art methods. Our new methods, along with asynchronous baselines, are implemented in a distributed framework which will be open sourced along with this publication.