iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients
This work addresses a key bottleneck in neural architecture search for researchers and practitioners, offering an incremental improvement over DARTS with better scalability and quality.
The paper tackles the challenge of hypergradient computation in Differentiable Architecture Search (DARTS) by using the implicit function theorem to make it path-agnostic, and proposes iDARTS with stochastic approximation to reduce computational requirements, leading to architectures that outperform baseline methods by large margins in experiments on NAS benchmark search spaces.
\textit{Differentiable ARchiTecture Search} (DARTS) has recently become the mainstream of neural architecture search (NAS) due to its efficiency and simplicity. With a gradient-based bi-level optimization, DARTS alternately optimizes the inner model weights and the outer architecture parameter in a weight-sharing supernet. A key challenge to the scalability and quality of the learned architectures is the need for differentiating through the inner-loop optimisation. While much has been discussed about several potentially fatal factors in DARTS, the architecture gradient, a.k.a. hypergradient, has received less attention. In this paper, we tackle the hypergradient computation in DARTS based on the implicit function theorem, making it only depends on the obtained solution to the inner-loop optimization and agnostic to the optimization path. To further reduce the computational requirements, we formulate a stochastic hypergradient approximation for differentiable NAS, and theoretically show that the architecture optimization with the proposed method, named iDARTS, is expected to converge to a stationary point. Comprehensive experiments on two NAS benchmark search spaces and the common NAS search space verify the effectiveness of our proposed method. It leads to architectures outperforming, with large margins, those learned by the baseline methods.