BN-NAS: Neural Architecture Search with Batch Normalization
This work addresses the high computational cost of neural architecture search for researchers and practitioners, offering a significant speed-up but is incremental as it builds on existing NAS methods.
The paper tackles the problem of accelerating neural architecture search by introducing BN-NAS, which uses a Batch Normalization-based indicator to predict subnet performance early and trains only BN parameters during supernet training, reducing supernet training time by over 10 times and subnet evaluation time by over 600,000 times without accuracy loss.
We present BN-NAS, neural architecture search with Batch Normalization (BN-NAS), to accelerate neural architecture search (NAS). BN-NAS can significantly reduce the time required by model training and evaluation in NAS. Specifically, for fast evaluation, we propose a BN-based indicator for predicting subnet performance at a very early training stage. The BN-based indicator further facilitates us to improve the training efficiency by only training the BN parameters during the supernet training. This is based on our observation that training the whole supernet is not necessary while training only BN parameters accelerates network convergence for network architecture search. Extensive experiments show that our method can significantly shorten the time of training supernet by more than 10 times and shorten the time of evaluating subnets by more than 600,000 times without losing accuracy.