RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search
This work addresses the need for resource-efficient neural architectures in real-world applications, but it is incremental as it builds on existing gradient-based NAS methods.
The authors tackled the problem of designing efficient neural architectures under resource constraints by extending a gradient-based Neural Architecture Search (NAS) method to support multiple resource costs, achieving specific performance on CIFAR-10 classification.
Early neural network architectures were designed by so-called "grad student descent". Since then, the field of Neural Architecture Search (NAS) has developed with the goal of algorithmically designing architectures tailored for a dataset of interest. Recently, gradient-based NAS approaches have been created to rapidly perform the search. Gradient-based approaches impose more structure on the search, compared to alternative NAS methods, enabling faster search phase optimization. In the real-world, neural architecture performance is measured by more than just high accuracy. There is increasing need for efficient neural architectures, where resources such as model size or latency must also be considered. Gradient-based NAS is also suitable for such multi-objective optimization. In this work we extend a popular gradient-based NAS method to support one or more resource costs. We then perform in-depth analysis on the discovery of architectures satisfying single-resource constraints for classification of CIFAR-10.