WPNAS: Neural Architecture Search by jointly using Weight Sharing and Predictor
This work addresses the challenge of efficient neural architecture search for machine learning practitioners, but it is incremental as it combines existing techniques.
The authors tackled the problem of neural architecture search by proposing WPNAS, a method that jointly uses weight sharing and a predictor to evaluate architectures, achieving state-of-the-art performance on datasets like CIFAR-10, CIFAR-100, and ImageNet.
Weight sharing based and predictor based methods are two major types of fast neural architecture search methods. In this paper, we propose to jointly use weight sharing and predictor in a unified framework. First, we construct a SuperNet in a weight-sharing way and probabilisticly sample architectures from the SuperNet. To increase the correctness of the evaluation of architectures, besides direct evaluation using the inherited weights, we further apply a few-shot predictor to assess the architecture on the other hand. The final evaluation of the architecture is the combination of direct evaluation, the prediction from the predictor and the cost of the architecture. We regard the evaluation as a reward and apply a self-critical policy gradient approach to update the architecture probabilities. To further reduce the side effects of weight sharing, we propose a weakly weight sharing method by introducing another HyperNet. We conduct experiments on datasets including CIFAR-10, CIFAR-100 and ImageNet under NATS-Bench, DARTS and MobileNet search space. The proposed WPNAS method achieves state-of-the-art performance on these datasets.