Transferrable Surrogates in Expressive Neural Architecture Search Spaces
This addresses the efficiency problem in neural architecture search for researchers and practitioners, though it appears incremental as it builds on existing surrogate and grammar-based methods.
The paper tackles the challenge of efficiently searching expressive neural architecture spaces by developing transferable surrogate models that predict architecture performance across datasets, achieving significant speed-ups and better final performances.
Neural architecture search (NAS) faces a challenge in balancing the exploration of expressive, broad search spaces that enable architectural innovation with the need for efficient evaluation of architectures to effectively search such spaces. We investigate surrogate model training for improving search in highly expressive NAS search spaces based on context-free grammars. We show that i) surrogate models trained either using zero-cost-proxy metrics and neural graph features (GRAF) or by fine-tuning an off-the-shelf LM have high predictive power for the performance of architectures both within and across datasets, ii) these surrogates can be used to filter out bad architectures when searching on novel datasets, thereby significantly speeding up search and achieving better final performances, and iii) the surrogates can be further used directly as the search objective for huge speed-ups.