Proxyless Neural Architecture Adaptation for Supervised Learning and Self-Supervised Learning
This addresses the problem of inefficient and non-reproducible architecture adaptation in neural network design, though it appears incremental as it builds upon existing NAT methods.
The paper tackles the issues of reproducibility and efficiency in Neural Architecture Adaptation by proposing a proxyless method that works for both supervised and self-supervised learning, showing stable performance and outperforming NAT on datasets like CIFAR-10 and Tiny Imagenet.
Recently, Neural Architecture Search (NAS) methods have been introduced and show impressive performance on many benchmarks. Among those NAS studies, Neural Architecture Transformer (NAT) aims to adapt the given neural architecture to improve performance while maintaining computational costs. However, NAT lacks reproducibility and it requires an additional architecture adaptation process before network weight training. In this paper, we propose proxyless neural architecture adaptation that is reproducible and efficient. Our method can be applied to both supervised learning and self-supervised learning. The proposed method shows stable performance on various architectures. Extensive reproducibility experiments on two datasets, i.e., CIFAR-10 and Tiny Imagenet, present that the proposed method definitely outperforms NAT and is applicable to other models and datasets.