DrNAS: Dirichlet Neural Architecture Search
It addresses the challenge of efficient and scalable NAS for machine learning practitioners, offering a novel approach that reduces memory consumption and bridges the gap between search and evaluation phases.
This paper tackles the problem of neural architecture search (NAS) by proposing DrNAS, a differentiable method that formulates NAS as a distribution learning problem using Dirichlet distributions, achieving a test error of 2.46% on CIFAR-10 and 23.7% on ImageNet under mobile settings.
This paper proposes a novel differentiable architecture search method by formulating it into a distribution learning problem. We treat the continuously relaxed architecture mixing weight as random variables, modeled by Dirichlet distribution. With recently developed pathwise derivatives, the Dirichlet parameters can be easily optimized with gradient-based optimizer in an end-to-end manner. This formulation improves the generalization ability and induces stochasticity that naturally encourages exploration in the search space. Furthermore, to alleviate the large memory consumption of differentiable NAS, we propose a simple yet effective progressive learning scheme that enables searching directly on large-scale tasks, eliminating the gap between search and evaluation phases. Extensive experiments demonstrate the effectiveness of our method. Specifically, we obtain a test error of 2.46% for CIFAR-10, 23.7% for ImageNet under the mobile setting. On NAS-Bench-201, we also achieve state-of-the-art results on all three datasets and provide insights for the effective design of neural architecture search algorithms.