NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy
This work addresses the need for standardized and comprehensive evaluation in NAS research, benefiting researchers by enabling more generalizable and efficient algorithm development, though it is incremental as it builds on existing benchmarks.
The paper tackles the problem of inconsistent and limited neural architecture search (NAS) benchmarks by analyzing popular NAS algorithms across 25 search space-dataset combinations, finding that conclusions from few benchmarks do not generalize, and introduces NAS-Bench-Suite, a unified collection of benchmarks to facilitate reproducible and rapid NAS research.
The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench-201, has significantly lowered the computational overhead for conducting scientific research in neural architecture search (NAS). Although they have been widely adopted and used to tune real-world NAS algorithms, these benchmarks are limited to small search spaces and focus solely on image classification. Recently, several new NAS benchmarks have been introduced that cover significantly larger search spaces over a wide range of tasks, including object detection, speech recognition, and natural language processing. However, substantial differences among these NAS benchmarks have so far prevented their widespread adoption, limiting researchers to using just a few benchmarks. In this work, we present an in-depth analysis of popular NAS algorithms and performance prediction methods across 25 different combinations of search spaces and datasets, finding that many conclusions drawn from a few NAS benchmarks do not generalize to other benchmarks. To help remedy this problem, we introduce NAS-Bench-Suite, a comprehensive and extensible collection of NAS benchmarks, accessible through a unified interface, created with the aim to facilitate reproducible, generalizable, and rapid NAS research. Our code is available at https://github.com/automl/naslib.