Enabling NAS with Automated Super-Network Generation
This addresses a bottleneck for researchers and practitioners in machine learning by enabling more efficient NAS adoption, though it is incremental as it builds on existing NAS techniques.
The paper tackles the challenge of constructing super-networks for arbitrary architectures in Neural Architecture Search (NAS) by introducing BootstrapNAS, a software framework that automatically generates super-networks from pre-trained models, resulting in subnetworks that significantly outperform the given pre-trained model.
Recent Neural Architecture Search (NAS) solutions have produced impressive results training super-networks and then deriving subnetworks, a.k.a. child models that outperform expert-crafted models from a pre-defined search space. Efficient and robust subnetworks can be selected for resource-constrained edge devices, allowing them to perform well in the wild. However, constructing super-networks for arbitrary architectures is still a challenge that often prevents the adoption of these approaches. To address this challenge, we present BootstrapNAS, a software framework for automatic generation of super-networks for NAS. BootstrapNAS takes a pre-trained model from a popular architecture, e.g., ResNet- 50, or from a valid custom design, and automatically creates a super-network out of it, then uses state-of-the-art NAS techniques to train the super-network, resulting in subnetworks that significantly outperform the given pre-trained model. We demonstrate the solution by generating super-networks from arbitrary model repositories and make available the resulting super-networks for reproducibility of the results.