NAS-Bench-101: Towards Reproducible Neural Architecture Search
This work addresses the barrier-to-entry for researchers without large-scale computation by providing a reproducible benchmark for NAS, though it is incremental as it builds on existing NAS methods.
The authors tackled the problem of high computational cost and lack of reproducibility in neural architecture search (NAS) by introducing NAS-Bench-101, a public dataset of 423k unique convolutional architectures with over 5 million trained models on CIFAR-10, enabling rapid evaluation in milliseconds.
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build NAS-Bench-101, we carefully constructed a compact, yet expressive, search space, exploiting graph isomorphisms to identify 423k unique convolutional architectures. We trained and evaluated all of these architectures multiple times on CIFAR-10 and compiled the results into a large dataset of over 5 million trained models. This allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset. We demonstrate its utility by analyzing the dataset as a whole and by benchmarking a range of architecture optimization algorithms.