NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization
This work addresses efficiency and performance limitations in NAS for researchers and practitioners in machine learning, offering incremental improvements over existing methods.
The paper tackles the problem of unbalanced time reduction and limited performance in neural architecture search (NAS) by introducing NetAdaptV2, which reduces total search time by up to 5.8x on ImageNet and 2.4x on NYU Depth V2 while discovering DNNs with better accuracy-latency trade-offs, outperforming NAS-discovered MobileNetV3 by 1.8% higher top-1 accuracy at the same latency.
Neural architecture search (NAS) typically consists of three main steps: training a super-network, training and evaluating sampled deep neural networks (DNNs), and training the discovered DNN. Most of the existing efforts speed up some steps at the cost of a significant slowdown of other steps or sacrificing the support of non-differentiable search metrics. The unbalanced reduction in the time spent per step limits the total search time reduction, and the inability to support non-differentiable search metrics limits the performance of discovered DNNs. In this paper, we present NetAdaptV2 with three innovations to better balance the time spent for each step while supporting non-differentiable search metrics. First, we propose channel-level bypass connections that merge network depth and layer width into a single search dimension to reduce the time for training and evaluating sampled DNNs. Second, ordered dropout is proposed to train multiple DNNs in a single forward-backward pass to decrease the time for training a super-network. Third, we propose the multi-layer coordinate descent optimizer that considers the interplay of multiple layers in each iteration of optimization to improve the performance of discovered DNNs while supporting non-differentiable search metrics. With these innovations, NetAdaptV2 reduces the total search time by up to $5.8\times$ on ImageNet and $2.4\times$ on NYU Depth V2, respectively, and discovers DNNs with better accuracy-latency/accuracy-MAC trade-offs than state-of-the-art NAS works. Moreover, the discovered DNN outperforms NAS-discovered MobileNetV3 by 1.8% higher top-1 accuracy with the same latency. The project website is http://netadapt.mit.edu.