Differentiable Neural Architecture Transformation for Reproducible Architecture Improvement
This addresses reproducibility issues in neural architecture improvement for researchers and practitioners, though it appears incremental as it builds directly on NAT.
The paper tackles the lack of reproducibility in Neural Architecture Transformer (NAT) methods by proposing a differentiable neural architecture transformation approach, which shows stable performance and outperforms NAT on datasets like CIFAR-10 and Tiny Imagenet.
Recently, Neural Architecture Search (NAS) methods are introduced and show impressive performance on many benchmarks. Among those NAS studies, Neural Architecture Transformer (NAT) aims to improve the given neural architecture to have better performance while maintaining computational costs. However, NAT has limitations about a lack of reproducibility. In this paper, we propose differentiable neural architecture transformation that is reproducible and efficient. The proposed method shows stable performance on various architectures. Extensive reproducibility experiments on two datasets, i.e., CIFAR-10 and Tiny Imagenet, present that the proposed method definitely outperforms NAT and be applicable to other models and datasets.