Efficacy of Neural Prediction-Based Zero-Shot NAS
This work addresses a domain-specific problem for researchers and practitioners in automated machine learning by improving zero-shot NAS generalizability across search spaces, though it is incremental as it builds on existing prediction-based and handcrafted NAS methods.
The paper tackles the limitation of existing prediction-based Neural Architecture Search (NAS) methods in evaluating architecture performance across different search spaces by proposing a novel zero-shot NAS approach using Fourier sum of sines encoding for convolutional kernels and an MLP for ranking. The method achieves higher correlation on the NAS-Bench-201 dataset and shows improved convergence and transferability across multiple NAS benchmarks.
In prediction-based Neural Architecture Search (NAS), performance indicators derived from graph convolutional networks have shown remarkable success. These indicators, achieved by representing feed-forward structures as component graphs through one-hot encoding, face a limitation: their inability to evaluate architecture performance across varying search spaces. In contrast, handcrafted performance indicators (zero-shot NAS), which use the same architecture with random initialization, can generalize across multiple search spaces. Addressing this limitation, we propose a novel approach for zero-shot NAS using deep learning. Our method employs Fourier sum of sines encoding for convolutional kernels, enabling the construction of a computational feed-forward graph with a structure similar to the architecture under evaluation. These encodings are learnable and offer a comprehensive view of the architecture's topological information. An accompanying multi-layer perceptron (MLP) then ranks these architectures based on their encodings. Experimental results show that our approach surpasses previous methods using graph convolutional networks in terms of correlation on the NAS-Bench-201 dataset and exhibits a higher convergence rate. Moreover, our extracted feature representation trained on each NAS benchmark is transferable to other NAS benchmarks, showing promising generalizability across multiple search spaces. The code is available at: https://github.com/minh1409/DFT-NPZS-NAS