MS-DARTS: Mean-Shift Based Differentiable Architecture Search
This addresses a specific bottleneck in Auto-ML for researchers and practitioners by improving the stability and accuracy of a widely-used NAS method, though it is incremental as it builds directly on DARTS.
The paper tackles the stabilization issues in Differentiable Architecture Search (DARTS) during discretization, which cause performance decline, by proposing MS-DARTS using mean-shift based sampling and perturbation to smooth the loss landscape, resulting in higher performance on CIFAR-10, CIFAR-100, and ImageNet with reduced search cost.
Differentiable Architecture Search (DARTS) is an effective continuous relaxation-based network architecture search (NAS) method with low search cost. It has attracted significant attentions in Auto-ML research and becomes one of the most useful paradigms in NAS. Although DARTS can produce superior efficiency over traditional NAS approaches with better control of complex parameters, oftentimes it suffers from stabilization issues in producing deteriorating architectures when discretizing the continuous architecture. We observed considerable loss of validity causing dramatic decline in performance at this final discretization step of DARTS. To address this issue, we propose a Mean-Shift based DARTS (MS-DARTS) to improve stability based on sampling and perturbation. Our approach can improve bot the stability and accuracy of DARTS, by smoothing the loss landscape and sampling architecture parameters within a suitable bandwidth. We investigate the convergence of our mean-shift approach, together with the effects of bandwidth selection that affects stability and accuracy. Evaluations performed on CIFAR-10, CIFAR-100, and ImageNet show that MS-DARTS archives higher performance over other state-of-the-art NAS methods with reduced search cost.