GreenFactory: Ensembling Zero-Cost Proxies to Estimate Performance of Neural Networks
This addresses the resource-intensive bottleneck of training-based evaluation in NAS for researchers and practitioners, though it is incremental as it builds on existing zero-cost proxies.
The paper tackles the problem of efficiently estimating neural network performance in Neural Architecture Search by proposing GreenFactory, an ensemble of zero-cost proxies that predicts test accuracy without training, achieving high Kendall correlations (e.g., 0.907-0.945 on NATS-Bench datasets).
Determining the performance of a Deep Neural Network during Neural Architecture Search processes is essential for identifying optimal architectures and hyperparameters. Traditionally, this process requires training and evaluation of each network, which is time-consuming and resource-intensive. Zero-cost proxies estimate performance without training, serving as an alternative to traditional training. However, recent proxies often lack generalization across diverse scenarios and provide only relative rankings rather than predicted accuracies. To address these limitations, we propose GreenFactory, an ensemble of zero-cost proxies that leverages a random forest regressor to combine multiple predictors' strengths and directly predict model test accuracy. We evaluate GreenFactory on NATS-Bench, achieving robust results across multiple datasets. Specifically, GreenFactory achieves high Kendall correlations on NATS-Bench-SSS, indicating substantial agreement between its predicted scores and actual performance: 0.907 for CIFAR-10, 0.945 for CIFAR-100, and 0.920 for ImageNet-16-120. Similarly, on NATS-Bench-TSS, we achieve correlations of 0.921 for CIFAR-10, 0.929 for CIFAR-100, and 0.908 for ImageNet-16-120, showcasing its reliability in both search spaces.