Neural Architecture Search: Two Constant Shared Weights Initialisations
This work addresses the computational bottleneck in NAS for researchers and practitioners by providing a faster, more efficient evaluation method, though it is incremental as it builds on existing zero-cost metric approaches.
The paper tackles the problem of efficiently evaluating neural architectures in neural architecture search (NAS) by introducing epsinas, a zero-cost metric that uses two constant shared weight initializations and output statistics to predict trained accuracy without training. The method requires no data labels, operates on a single minibatch, eliminates gradient computation, and evaluates networks in a fraction of a GPU second, showing strong correlation with accuracy across image classification and language tasks on standard benchmarks.
In the last decade, zero-cost metrics have gained prominence in neural architecture search (NAS) due to their ability to evaluate architectures without training. These metrics are significantly faster and less computationally expensive than traditional NAS methods and provide insights into neural architectures' internal workings. This paper introduces epsinas, a novel zero-cost NAS metric that assesses architecture potential using two constant shared weight initialisations and the statistics of their outputs. We show that the dispersion of raw outputs, normalised by their average magnitude, strongly correlates with trained accuracy. This effect holds across image classification and language tasks on NAS-Bench-101, NAS-Bench-201, and NAS-Bench-NLP. Our method requires no data labels, operates on a single minibatch, and eliminates the need for gradient computation, making it independent of training hyperparameters, loss metrics, and human annotations. It evaluates a network in a fraction of a GPU second and integrates seamlessly into existing NAS frameworks. The code supporting this study can be found on GitHub at https://github.com/egracheva/epsinas.