TG-NAS: Generalizable Zero-Cost Proxies with Operator Description Embedding and Graph Learning for Efficient Neural Architecture Search
This addresses the need for efficient and generalizable NAS methods for researchers and practitioners, though it is incremental as it builds on existing zero-shot NAS and model-based proxy approaches.
The paper tackles the problem of inefficient and poorly generalizing zero-cost proxies in Neural Architecture Search (NAS) by proposing TG-NAS, a universal model-based proxy that combines Transformer-based operator embeddings and Graph Convolutional Networks to predict architecture performance without retraining. The result shows TG-NAS achieves up to 300x improved search efficiency and discovers architectures with 93.75% CIFAR-10 accuracy on NAS-Bench-201 and 74.9% ImageNet top-1 accuracy on the DARTS space.
Neural Architecture Search (NAS) is a powerful technique for discovering high-performing CNN architectures, but most existing methods rely on costly training or extensive sampling. Zero-shot NAS offers a training-free alternative by using proxies to predict architecture performance. However, existing proxies are often suboptimal -- frequently outperformed by simple metrics like parameter count or FLOPs -- and they generalize poorly across different search spaces. Moreover, current model-based proxies struggle to adapt to new operators without access to ground-truth accuracy, limiting their transferability. We propose TG-NAS, a universal, model-based zero-cost (ZC) proxy that combines a Transformer-based operator embedding generator with a Graph Convolutional Network (GCN) to predict architecture performance. Unlike prior model-based predictors, TG-NAS requires no retraining and generalizes across arbitrary search spaces. It serves as a standalone ZC proxy with strong data efficiency, robustness, and cross-space consistency. Extensive evaluations across diverse NAS benchmarks demonstrate TG-NAS's superior rank correlation and generalizability compared to existing proxies. Additionally, it improves search efficiency by up to 300x and discovers architectures achieving 93.75% CIFAR-10 accuracy on NAS-Bench-201 and 74.9% ImageNet top-1 accuracy on the DARTS space, establishing TG-NAS as a promising foundation for efficient, generalizable NAS.