DARTS+: Improved Differentiable Architecture Search with Early Stopping
This addresses a critical instability problem in neural architecture search for researchers and practitioners, though it is an incremental improvement over existing DARTS variants.
The paper tackles the performance collapse issue in Differentiable Architecture Search (DARTS) by proposing DARTS+, which uses early stopping to prevent overfitting, achieving test errors of 2.32% on CIFAR10, 14.87% on CIFAR100, and 23.7% on ImageNet.
Recently, there has been a growing interest in automating the process of neural architecture design, and the Differentiable Architecture Search (DARTS) method makes the process available within a few GPU days. However, the performance of DARTS is often observed to collapse when the number of search epochs becomes large. Meanwhile, lots of "{\em skip-connect}s" are found in the selected architectures. In this paper, we claim that the cause of the collapse is that there exists overfitting in the optimization of DARTS. Therefore, we propose a simple and effective algorithm, named "DARTS+", to avoid the collapse and improve the original DARTS, by "early stopping" the search procedure when meeting a certain criterion. We also conduct comprehensive experiments on benchmark datasets and different search spaces and show the effectiveness of our DARTS+ algorithm, and DARTS+ achieves $2.32\%$ test error on CIFAR10, $14.87\%$ on CIFAR100, and $23.7\%$ on ImageNet. We further remark that the idea of "early stopping" is implicitly included in some existing DARTS variants by manually setting a small number of search epochs, while we give an {\em explicit} criterion for "early stopping".