Neural Architecture Search using Covariance Matrix Adaptation Evolution Strategy
This work addresses the problem of long search times in neural architecture search for researchers and practitioners, representing an incremental improvement over previous evolution-based methods.
The paper tackled the high computational cost of evolution-based neural architecture search by proposing CMANAS, which uses CMA-ES and achieves top-1 test accuracies of 97.44% on CIFAR-10 in 0.45 GPU day and 83.24% on CIFAR-100 in 0.6 GPU day, with transferred architectures reaching top-5 accuracies up to 92.6% on ImageNet.
Evolution-based neural architecture search requires high computational resources, resulting in long search time. In this work, we propose a framework of applying the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to the neural architecture search problem called CMANAS, which achieves better results than previous evolution-based methods while reducing the search time significantly. The architectures are modelled using a normal distribution, which is updated using CMA-ES based on the fitness of the sampled population. We used the accuracy of a trained one shot model (OSM) on the validation data as a prediction of the fitness of an individual architecture to reduce the search time. We also used an architecture-fitness table (AF table) for keeping record of the already evaluated architecture, thus further reducing the search time. CMANAS finished the architecture search on CIFAR-10 with the top-1 test accuracy of 97.44% in 0.45 GPU day and on CIFAR-100 with the top-1 test accuracy of 83.24% for 0.6 GPU day on a single GPU. The top architectures from the searches on CIFAR-10 and CIFAR-100 were then transferred to ImageNet, achieving the top-5 accuracy of 92.6% and 92.1%, respectively.