Angle-based Search Space Shrinking for Neural Architecture Search
This work addresses the problem of computational inefficiency in NAS for researchers and practitioners, though it is incremental as it builds on existing NAS approaches.
The paper tackles the challenge of efficiently shrinking the search space in Neural Architecture Search (NAS) by introducing an angle-based metric to drop unpromising candidates, resulting in dramatic enhancement of existing NAS methods with faster convergence and improved stability.
In this work, we present a simple and general search space shrinking method, called Angle-Based search space Shrinking (ABS), for Neural Architecture Search (NAS). Our approach progressively simplifies the original search space by dropping unpromising candidates, thus can reduce difficulties for existing NAS methods to find superior architectures. In particular, we propose an angle-based metric to guide the shrinking process. We provide comprehensive evidences showing that, in weight-sharing supernet, the proposed metric is more stable and accurate than accuracy-based and magnitude-based metrics to predict the capability of child models. We also show that the angle-based metric can converge fast while training supernet, enabling us to get promising shrunk search spaces efficiently. ABS can easily apply to most of NAS approaches (e.g. SPOS, FairNAS, ProxylessNAS, DARTS and PDARTS). Comprehensive experiments show that ABS can dramatically enhance existing NAS approaches by providing a promising shrunk search space.