ReNAS:Relativistic Evaluation of Neural Architecture Search
This addresses the computational bottleneck in NAS for researchers and practitioners by providing a more efficient evaluation method, though it is incremental as it builds on existing NAS frameworks.
The paper tackles the problem of inaccurate performance estimation in Neural Architecture Search (NAS) by proposing a relativistic evaluation scheme, ReNAS, which predicts which architecture performs better rather than absolute performance, achieving higher accuracies than state-of-the-art methods on NAS-Bench datasets with only 0.1% of the search space sampled.
An effective and efficient architecture performance evaluation scheme is essential for the success of Neural Architecture Search (NAS). To save computational cost, most of existing NAS algorithms often train and evaluate intermediate neural architectures on a small proxy dataset with limited training epochs. But it is difficult to expect an accurate performance estimation of an architecture in such a coarse evaluation way. This paper advocates a new neural architecture evaluation scheme, which aims to determine which architecture would perform better instead of accurately predict the absolute architecture performance. Therefore, we propose a \textbf{relativistic} architecture performance predictor in NAS (ReNAS). We encode neural architectures into feature tensors, and further refining the representations with the predictor. The proposed relativistic performance predictor can be deployed in discrete searching methods to search for the desired architectures without additional evaluation. Experimental results on NAS-Bench-101 dataset suggests that, sampling 424 ($0.1\%$ of the entire search space) neural architectures and their corresponding validation performance is already enough for learning an accurate architecture performance predictor. The accuracies of our searched neural architectures on NAS-Bench-101 and NAS-Bench-201 datasets are higher than that of the state-of-the-art methods and show the priority of the proposed method.