Evolving Search Space for Neural Architecture Search
This work provides a more automated and flexible approach to neural architecture design, which is significant for researchers and practitioners in deep learning seeking to reduce manual design effort and improve model performance.
This paper tackles the problem of limited search space in Neural Architecture Search (NAS) by introducing Neural Search-space Evolution (NSE), an iterative scheme that optimizes search space subsets. This method achieved a state-of-the-art 77.3% top-1 retrain accuracy on ImageNet with 333M FLOPs for auto-generated architectures without knowledge distillation or weight pruning, and 77.9% Top-1 accuracy under latency constraints.
The automation of neural architecture design has been a coveted alternative to human experts. Recent works have small search space, which is easier to optimize but has a limited upper bound of the optimal solution. Extra human design is needed for those methods to propose a more suitable space with respect to the specific task and algorithm capacity. To further enhance the degree of automation for neural architecture search, we present a Neural Search-space Evolution (NSE) scheme that iteratively amplifies the results from the previous effort by maintaining an optimized search space subset. This design minimizes the necessity of a well-designed search space. We further extend the flexibility of obtainable architectures by introducing a learnable multi-branch setting. By employing the proposed method, a consistent performance gain is achieved during a progressive search over upcoming search spaces. We achieve 77.3% top-1 retrain accuracy on ImageNet with 333M FLOPs, which yielded a state-of-the-art performance among previous auto-generated architectures that do not involve knowledge distillation or weight pruning. When the latency constraint is adopted, our result also performs better than the previous best-performing mobile models with a 77.9% Top-1 retrain accuracy.