MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation
This addresses efficiency and reliability issues in NAS for machine learning practitioners, offering an incremental improvement over existing methods like DARTS.
The paper tackles the problem of gradient errors causing suboptimal convergence in Neural Architecture Search (NAS) by proposing MiLeNAS, a mixed-level reformulation that achieves lower validation errors and higher accuracies than bilevel optimization methods, with search completed in around 5 hours.
Many recently proposed methods for Neural Architecture Search (NAS) can be formulated as bilevel optimization. For efficient implementation, its solution requires approximations of second-order methods. In this paper, we demonstrate that gradient errors caused by such approximations lead to suboptimality, in the sense that the optimization procedure fails to converge to a (locally) optimal solution. To remedy this, this paper proposes \mldas, a mixed-level reformulation for NAS that can be optimized efficiently and reliably. It is shown that even when using a simple first-order method on the mixed-level formulation, \mldas\ can achieve a lower validation error for NAS problems. Consequently, architectures obtained by our method achieve consistently higher accuracies than those obtained from bilevel optimization. Moreover, \mldas\ proposes a framework beyond DARTS. It is upgraded via model size-based search and early stopping strategies to complete the search process in around 5 hours. Extensive experiments within the convolutional architecture search space validate the effectiveness of our approach.