Towards Self-supervised and Weight-preserving Neural Architecture Search
This work addresses the fussy procedures and supervised learning limitations in NAS for real-world applications, offering an incremental improvement by extending current frameworks with self-supervision and weight retention.
The paper tackles the cumbersome deployment of neural architecture search (NAS) by proposing a self-supervised and weight-preserving framework that simplifies the workflow to a one-stage, proxy-free procedure, achieving state-of-the-art accuracy on CIFAR-10, CIFAR-100, and ImageNet without manual labels and showing that using discovered weights as initialization outperforms random initialization and two-stage pre-training in semi-supervised learning.
Neural architecture search (NAS) algorithms save tremendous labor from human experts. Recent advancements further reduce the computational overhead to an affordable level. However, it is still cumbersome to deploy the NAS techniques in real-world applications due to the fussy procedures and the supervised learning paradigm. In this work, we propose the self-supervised and weight-preserving neural architecture search (SSWP-NAS) as an extension of the current NAS framework by allowing the self-supervision and retaining the concomitant weights discovered during the search stage. As such, we simplify the workflow of NAS to a one-stage and proxy-free procedure. Experiments show that the architectures searched by the proposed framework achieve state-of-the-art accuracy on CIFAR-10, CIFAR-100, and ImageNet datasets without using manual labels. Moreover, we show that employing the concomitant weights as initialization consistently outperforms the random initialization and the two-stage weight pre-training method by a clear margin under semi-supervised learning scenarios. Codes are publicly available at https://github.com/LzVv123456/SSWP-NAS.