Homogeneous Architecture Augmentation for Neural Predictor
This work addresses a bottleneck in NAS for researchers by improving neural predictors with data augmentation, though it appears incremental as it builds on existing methods.
The paper tackles the shortage of annotated DNN architectures for training neural predictors in Neural Architecture Search (NAS) by proposing HAAP, which uses homogeneous architecture augmentation and one-hot encoding to generate sufficient training data, resulting in outperforming state-of-the-art methods with much less training data on NAS-Benchmark-101 and NAS-Bench-201 datasets.
Neural Architecture Search (NAS) can automatically design well-performed architectures of Deep Neural Networks (DNNs) for the tasks at hand. However, one bottleneck of NAS is the prohibitively computational cost largely due to the expensive performance evaluation. The neural predictors can directly estimate the performance without any training of the DNNs to be evaluated, thus have drawn increasing attention from researchers. Despite their popularity, they also suffer a severe limitation: the shortage of annotated DNN architectures for effectively training the neural predictors. In this paper, we proposed Homogeneous Architecture Augmentation for Neural Predictor (HAAP) of DNN architectures to address the issue aforementioned. Specifically, a homogeneous architecture augmentation algorithm is proposed in HAAP to generate sufficient training data taking the use of homogeneous representation. Furthermore, the one-hot encoding strategy is introduced into HAAP to make the representation of DNN architectures more effective. The experiments have been conducted on both NAS-Benchmark-101 and NAS-Bench-201 dataset. The experimental results demonstrate that the proposed HAAP algorithm outperforms the state of the arts compared, yet with much less training data. In addition, the ablation studies on both benchmark datasets have also shown the universality of the homogeneous architecture augmentation.