Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning
This work addresses the computational bottleneck in NAS for researchers and practitioners, offering an incremental improvement through pretraining and metric learning.
The paper tackles the high computational cost of neural architecture search (NAS) by proposing a pretraining scheme using locality-based self-supervised learning to improve architecture representations, resulting in enhanced performance in NAS tasks with specific gains such as reduced search time and improved accuracy.
Neural architecture search (NAS) has fostered various fields of machine learning. Despite its prominent dedications, many have criticized the intrinsic limitations of high computational cost. We aim to ameliorate this by proposing a pretraining scheme that can be generally applied to controller-based NAS. Our method, locality-based self-supervised classification task, leverages the structural similarity of network architectures to obtain good architecture representations. We incorporate our method into neural architecture optimization (NAO) to analyze the pretrained embeddings and its effectiveness and highlight that adding metric learning loss brings a favorable impact on NAS. Our code is available at \url{https://github.com/Multi-Objective-NAS/self-supervised-nas}.