Hierarchical Neural Architecture Search via Operator Clustering
This addresses a key limitation in automated neural architecture design for machine learning practitioners, offering a more robust method, though it is incremental as it builds on DARTS.
The paper tackles the poor generalization of gradient-based neural architecture search (DARTS) when search spaces change, by identifying two contributing factors and proposing a hierarchical search algorithm with operator clustering and optimization complexity match, achieving state-of-the-art accuracy on CIFAR-10, CIFAR-100, and ImageNet across five search space variants.
Recently, the efficiency of automatic neural architecture design has been significantly improved by gradient-based search methods such as DARTS. However, recent literature has brought doubt to the generalization ability of DARTS, arguing that DARTS performs poorly when the search space is changed, i.e, when different set of candidate operators are used. Regularization techniques such as early stopping have been proposed to partially solve this problem. In this paper, we tackle this problem from a different perspective by identifying two contributing factors to the collapse of DARTS when the search space changes: (1) the correlation of similar operators incurs unfavorable competition among them and makes their relative importance score unreliable and (2) the optimization complexity gap between the proxy search stage and the final training. Based on these findings, we propose a new hierarchical search algorithm. With its operator clustering and optimization complexity match, the algorithm can consistently find high-performance architecture across various search spaces. For all the five variants of the popular cell-based search spaces, the proposed algorithm always obtains state-of-the-art architecture with best accuracy on the CIFAR-10, CIFAR-100 and ImageNet over other well-established DARTS-alike algorithms. Code is available at https://github.com/susan0199/StacNAS.