Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator
This work addresses the high computational cost and inflexibility in neural architecture search for researchers and practitioners needing efficient deployment across diverse hardware.
The paper tackles the inefficiency of searching for multiple hardware-constrained neural architectures in one-shot NAS by proposing an architecture generator that produces architectures in one forward pass, reducing search time to 5 GPU hours for N constraints, which is 4N times faster than prior methods, while achieving 77.1% top-1 accuracy on ImageNet.
In one-shot NAS, sub-networks need to be searched from the supernet to meet different hardware constraints. However, the search cost is high and $N$ times of searches are needed for $N$ different constraints. In this work, we propose a novel search strategy called architecture generator to search sub-networks by generating them, so that the search process can be much more efficient and flexible. With the trained architecture generator, given target hardware constraints as the input, $N$ good architectures can be generated for $N$ constraints by just one forward pass without re-searching and supernet retraining. Moreover, we propose a novel single-path supernet, called unified supernet, to further improve search efficiency and reduce GPU memory consumption of the architecture generator. With the architecture generator and the unified supernet, we propose a flexible and efficient one-shot NAS framework, called Searching by Generating NAS (SGNAS). With the pre-trained supernt, the search time of SGNAS for $N$ different hardware constraints is only 5 GPU hours, which is $4N$ times faster than previous SOTA single-path methods. After training from scratch, the top1-accuracy of SGNAS on ImageNet is 77.1%, which is comparable with the SOTAs. The code is available at: https://github.com/eric8607242/SGNAS.