Discretization-Aware Architecture Search
This work addresses a specific bottleneck in NAS for researchers and practitioners by improving the accuracy of weight-sharing methods, though it is incremental as it builds on existing NAS frameworks.
The paper tackles the inaccuracy problem in neural architecture search (NAS) caused by discretization during the pruning of weak candidates from a super-network, and introduces discretization-aware architecture search (DA²S) which adds a loss term to align the super-network with the desired topology, reducing accuracy loss and showing superiority on standard image classification benchmarks, especially under imbalanced target network configurations.
The search cost of neural architecture search (NAS) has been largely reduced by weight-sharing methods. These methods optimize a super-network with all possible edges and operations, and determine the optimal sub-network by discretization, \textit{i.e.}, pruning off weak candidates. The discretization process, performed on either operations or edges, incurs significant inaccuracy and thus the quality of the final architecture is not guaranteed. This paper presents discretization-aware architecture search (DA\textsuperscript{2}S), with the core idea being adding a loss term to push the super-network towards the configuration of desired topology, so that the accuracy loss brought by discretization is largely alleviated. Experiments on standard image classification benchmarks demonstrate the superiority of our approach, in particular, under imbalanced target network configurations that were not studied before.