Learning to Rank Ace Neural Architectures via Normalized Discounted Cumulative Gain
This work addresses the challenge of identifying top architectures in NAS more efficiently, which is incremental as it builds on existing ranking methods by focusing on a specific metric and optimization approach.
The paper tackled the problem of efficiently ranking neural architectures in Neural Architecture Search (NAS) by proposing Normalized Discounted Cumulative Gain (NDCG) as a better metric than traditional ranking correlations, and introduced AceNAS, which optimizes NDCG with LambdaRank and uses weak labels for pre-training, resulting in up to 3.67% accuracy improvement and 8x reduction in search cost across benchmarks.
One of the key challenges in Neural Architecture Search (NAS) is to efficiently rank the performances of architectures. The mainstream assessment of performance rankers uses ranking correlations (e.g., Kendall's tau), which pay equal attention to the whole space. However, the optimization goal of NAS is identifying top architectures while paying less attention on other architectures in the search space. In this paper, we show both empirically and theoretically that Normalized Discounted Cumulative Gain (NDCG) is a better metric for rankers. Subsequently, we propose a new algorithm, AceNAS, which directly optimizes NDCG with LambdaRank. It also leverages weak labels produced by weight-sharing NAS to pre-train the ranker, so as to further reduce search cost. Extensive experiments on 12 NAS benchmarks and a large-scale search space demonstrate that our approach consistently outperforms SOTA NAS methods, with up to 3.67% accuracy improvement and 8x reduction on search cost.