Traditional and accelerated gradient descent for neural architecture search
This work addresses the computational efficiency problem in neural architecture search for machine learning researchers, offering an incremental improvement over existing methods.
The paper tackles neural architecture search by introducing two algorithms (NASGD and NASAGD) that optimize architectures using gradient descent on a semi-discrete space, achieving a 40x increase in analyzed architectures compared to hill climbing methods with similar computational resources and accuracy, such as a 4.06% error rate on CIFAR-10 in 12 hours on a single GPU.
In this paper we introduce two algorithms for neural architecture search (NASGD and NASAGD) following the theoretical work by two of the authors [5] which used the geometric structure of optimal transport to introduce the conceptual basis for new notions of traditional and accelerated gradient descent algorithms for the optimization of a function on a semi-discrete space. Our algorithms, which use the network morphism framework introduced in [2] as a baseline, can analyze forty times as many architectures as the hill climbing methods [2, 14] while using the same computational resources and time and achieving comparable levels of accuracy. For example, using NASGD on CIFAR-10, our method designs and trains networks with an error rate of 4.06 in only 12 hours on a single GPU.