Revisiting Training-free NAS Metrics: An Efficient Training-based Method
This work addresses the efficiency and accuracy trade-off in NAS for machine learning practitioners, offering an incremental improvement by reducing search cost while maintaining performance.
The paper tackles the problem of neural architecture search (NAS) by showing that recent training-free metrics rely heavily on parameter count, and proposes a lightweight training-based metric that achieves competitive top-1/top-5 error rates of 24.1%/7.1% on ImageNet in only 2.6 GPU hours.
Recent neural architecture search (NAS) works proposed training-free metrics to rank networks which largely reduced the search cost in NAS. In this paper, we revisit these training-free metrics and find that: (1) the number of parameters (\#Param), which is the most straightforward training-free metric, is overlooked in previous works but is surprisingly effective, (2) recent training-free metrics largely rely on the \#Param information to rank networks. Our experiments show that the performance of recent training-free metrics drops dramatically when the \#Param information is not available. Motivated by these observations, we argue that metrics less correlated with the \#Param are desired to provide additional information for NAS. We propose a light-weight training-based metric which has a weak correlation with the \#Param while achieving better performance than training-free metrics at a lower search cost. Specifically, on DARTS search space, our method completes searching directly on ImageNet in only 2.6 GPU hours and achieves a top-1/top-5 error rate of 24.1\%/7.1\%, which is competitive among state-of-the-art NAS methods. Codes are available at \url{https://github.com/taoyang1122/Revisit_TrainingFree_NAS}