RANK-NOSH: Efficient Predictor-Based Architecture Search via Non-Uniform Successive Halving
This work addresses efficiency issues in neural architecture search for researchers and practitioners, representing an incremental improvement over existing predictor-based methods.
The paper tackles the high computational cost of predictor-based neural architecture search by proposing a hierarchical scheduling algorithm that terminates underperforming architectures early, reducing the search budget by approximately 5x while achieving competitive or better performance than previous state-of-the-art methods.
Predictor-based algorithms have achieved remarkable performance in the Neural Architecture Search (NAS) tasks. However, these methods suffer from high computation costs, as training the performance predictor usually requires training and evaluating hundreds of architectures from scratch. Previous works along this line mainly focus on reducing the number of architectures required to fit the predictor. In this work, we tackle this challenge from a different perspective - improve search efficiency by cutting down the computation budget of architecture training. We propose NOn-uniform Successive Halving (NOSH), a hierarchical scheduling algorithm that terminates the training of underperforming architectures early to avoid wasting budget. To effectively leverage the non-uniform supervision signals produced by NOSH, we formulate predictor-based architecture search as learning to rank with pairwise comparisons. The resulting method - RANK-NOSH, reduces the search budget by ~5x while achieving competitive or even better performance than previous state-of-the-art predictor-based methods on various spaces and datasets.