FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
This work addresses a bottleneck in neural architecture search for efficient model design, benefiting researchers and practitioners in computer vision and mobile AI, though it is incremental as it builds upon existing DNAS methods.
The paper tackles the memory and computational inefficiency of Differentiable Neural Architecture Search (DNAS) by proposing DMaskingNAS, which expands the search space by up to 10^14 times and reduces search cost by up to 421x, resulting in FBNetV2 models that achieve state-of-the-art performance with up to 0.9% higher accuracy and 15-20% fewer FLOPs compared to prior architectures.
Differentiable Neural Architecture Search (DNAS) has demonstrated great success in designing state-of-the-art, efficient neural networks. However, DARTS-based DNAS's search space is small when compared to other search methods', since all candidate network layers must be explicitly instantiated in memory. To address this bottleneck, we propose a memory and computationally efficient DNAS variant: DMaskingNAS. This algorithm expands the search space by up to $10^{14}\times$ over conventional DNAS, supporting searches over spatial and channel dimensions that are otherwise prohibitively expensive: input resolution and number of filters. We propose a masking mechanism for feature map reuse, so that memory and computational costs stay nearly constant as the search space expands. Furthermore, we employ effective shape propagation to maximize per-FLOP or per-parameter accuracy. The searched FBNetV2s yield state-of-the-art performance when compared with all previous architectures. With up to 421$\times$ less search cost, DMaskingNAS finds models with 0.9% higher accuracy, 15% fewer FLOPs than MobileNetV3-Small; and with similar accuracy but 20% fewer FLOPs than Efficient-B0. Furthermore, our FBNetV2 outperforms MobileNetV3 by 2.6% in accuracy, with equivalent model size. FBNetV2 models are open-sourced at https://github.com/facebookresearch/mobile-vision.