CAP: A Context-Aware Neural Predictor for NAS
This work addresses the efficiency problem in NAS for researchers and practitioners by reducing the need for annotated data, though it is incremental as it builds on existing neural predictor methods.
The paper tackles the challenge of training neural predictors for neural architecture search (NAS) with limited annotated architectures, proposing a context-aware neural predictor (CAP) that uses contextual information and self-supervised tasks to achieve precise architecture ranking with only 172 annotated architectures in NAS-Bench-101.
Neural predictors are effective in boosting the time-consuming performance evaluation stage in neural architecture search (NAS), owing to their direct estimation of unseen architectures. Despite the effectiveness, training a powerful neural predictor with fewer annotated architectures remains a huge challenge. In this paper, we propose a context-aware neural predictor (CAP) which only needs a few annotated architectures for training based on the contextual information from the architectures. Specifically, the input architectures are encoded into graphs and the predictor infers the contextual structure around the nodes inside each graph. Then, enhanced by the proposed context-aware self-supervised task, the pre-trained predictor can obtain expressive and generalizable representations of architectures. Therefore, only a few annotated architectures are sufficient for training. Experimental results in different search spaces demonstrate the superior performance of CAP compared with state-of-the-art neural predictors. In particular, CAP can rank architectures precisely at the budget of only 172 annotated architectures in NAS-Bench-101. Moreover, CAP can help find promising architectures in both NAS-Bench-101 and DARTS search spaces on the CIFAR-10 dataset, serving as a useful navigator for NAS to explore the search space efficiently.