Exploring Complicated Search Spaces with Interleaving-Free Sampling
This work addresses a bottleneck in NAS for researchers and practitioners by enabling exploration of more complex architectures, though it is incremental as it builds upon existing weight-sharing approaches.
The paper tackles the problem of neural architecture search (NAS) in complicated search spaces with long-distance connections, where existing weight-sharing algorithms fail due to interleaved connections, and presents IF-NAS, a periodic sampling algorithm that avoids these connections and outperforms random sampling and previous methods by a significant margin.
The existing neural architecture search algorithms are mostly working on search spaces with short-distance connections. We argue that such designs, though safe and stable, obstacles the search algorithms from exploring more complicated scenarios. In this paper, we build the search algorithm upon a complicated search space with long-distance connections, and show that existing weight-sharing search algorithms mostly fail due to the existence of \textbf{interleaved connections}. Based on the observation, we present a simple yet effective algorithm named \textbf{IF-NAS}, where we perform a periodic sampling strategy to construct different sub-networks during the search procedure, avoiding the interleaved connections to emerge in any of them. In the proposed search space, IF-NAS outperform both random sampling and previous weight-sharing search algorithms by a significant margin. IF-NAS also generalizes to the micro cell-based spaces which are much easier. Our research emphasizes the importance of macro structure and we look forward to further efforts along this direction.