A Study of the Learning Progress in Neural Architecture Search Techniques
This work reveals that ENAS's success may stem from search space design rather than controller learning, suggesting one-shot architecture design as an efficient alternative, which is an incremental finding for the neural architecture search community.
The study investigated the learning progress of the controller in Efficient Neural Architecture Search (ENAS) and found that there was no observable improvement in the performance of generated architectures over training epochs, with consistent results across CIFAR-10, CIFAR-100, and two search spaces.
In neural architecture search, the structure of the neural network to best model a given dataset is determined by an automated search process. Efficient Neural Architecture Search (ENAS), proposed by Pham et al. (2018), has recently received considerable attention due to its ability to find excellent architectures within a comparably short search time. In this work, which is motivated by the quest to further improve the learning speed of architecture search, we evaluate the learning progress of the controller which generates the architectures in ENAS. We measure the progress by comparing the architectures generated by it at different controller training epochs, where architectures are evaluated after having re-trained them from scratch. As a surprising result, we find that the learning curves are completely flat, i.e., there is no observable progress of the controller in terms of the performance of its generated architectures. This observation is consistent across the CIFAR-10 and CIFAR-100 datasets and two different search spaces. We conclude that the high quality of the models generated by ENAS is a result of the search space design rather than the controller training, and our results indicate that one-shot architecture design is an efficient alternative to architecture search by ENAS.