AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search
This work addresses a key bottleneck in NAS for machine learning practitioners by improving the efficiency and accuracy of architecture selection without training, though it is incremental in combining existing proxy ideas.
The paper tackles the problem of weak correlation between training-free network architecture search (NAS) rankings and actual performance by proposing AZ-NAS, which ensembles novel zero-cost proxies to enhance this correlation, achieving state-of-the-art results on standard benchmarks with efficient runtime.
Training-free network architecture search (NAS) aims to discover high-performing networks with zero-cost proxies, capturing network characteristics related to the final performance. However, network rankings estimated by previous training-free NAS methods have shown weak correlations with the performance. To address this issue, we propose AZ-NAS, a novel approach that leverages the ensemble of various zero-cost proxies to enhance the correlation between a predicted ranking of networks and the ground truth substantially in terms of the performance. To achieve this, we introduce four novel zero-cost proxies that are complementary to each other, analyzing distinct traits of architectures in the views of expressivity, progressivity, trainability, and complexity. The proxy scores can be obtained simultaneously within a single forward and backward pass, making an overall NAS process highly efficient. In order to integrate the rankings predicted by our proxies effectively, we introduce a non-linear ranking aggregation method that highlights the networks highly-ranked consistently across all the proxies. Experimental results conclusively demonstrate the efficacy and efficiency of AZ-NAS, outperforming state-of-the-art methods on standard benchmarks, all while maintaining a reasonable runtime cost.