A Hardware-Aware Framework for Accelerating Neural Architecture Search Across Modalities
This work addresses the under-explored challenge of hardware-aware sub-network search in NAS, offering a solution for researchers and practitioners needing efficient multi-objective optimization, though it is incremental as it builds on existing NAS methods.
The paper tackles the problem of efficiently finding optimal sub-networks for different hardware configurations in Neural Architecture Search (NAS) by proposing a flexible framework that pairs evolutionary algorithms with lightly trained objective predictors, achieving accelerated architecture search across modalities like machine translation and image classification.
Recent advances in Neural Architecture Search (NAS) such as one-shot NAS offer the ability to extract specialized hardware-aware sub-network configurations from a task-specific super-network. While considerable effort has been employed towards improving the first stage, namely, the training of the super-network, the search for derivative high-performing sub-networks is still under-explored. Popular methods decouple the super-network training from the sub-network search and use performance predictors to reduce the computational burden of searching on different hardware platforms. We propose a flexible search framework that automatically and efficiently finds optimal sub-networks that are optimized for different performance metrics and hardware configurations. Specifically, we show how evolutionary algorithms can be paired with lightly trained objective predictors in an iterative cycle to accelerate architecture search in a multi-objective setting for various modalities including machine translation and image classification.