Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars
This work addresses the problem of designing expressive and scalable search spaces for neural architecture search, which is incremental as it builds on hierarchical NAS concepts.
The paper tackles the lack of a unifying framework for hierarchical neural architecture search spaces by introducing one based on context-free grammars, which generates spaces 100s of orders of magnitude larger than common ones and demonstrates superior search performance compared to existing NAS approaches.
The discovery of neural architectures from simple building blocks is a long-standing goal of Neural Architecture Search (NAS). Hierarchical search spaces are a promising step towards this goal but lack a unifying search space design framework and typically only search over some limited aspect of architectures. In this work, we introduce a unifying search space design framework based on context-free grammars that can naturally and compactly generate expressive hierarchical search spaces that are 100s of orders of magnitude larger than common spaces from the literature. By enhancing and using their properties, we effectively enable search over the complete architecture and can foster regularity. Further, we propose an efficient hierarchical kernel design for a Bayesian Optimization search strategy to efficiently search over such huge spaces. We demonstrate the versatility of our search space design framework and show that our search strategy can be superior to existing NAS approaches. Code is available at https://github.com/automl/hierarchical_nas_construction.