Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization
This addresses robustness problems in neural architecture search for researchers, but it is incremental as it builds on existing DARTS methods.
The paper tackles the robustness issues and performance discrepancies in Differentiable Architecture Search (DARTS) by identifying information bypass leakage during supernet training, and proposes a sparse-regularized approximation with a mixed-sparsity training scheme to eliminate this leakage. Experiments across multiple search spaces demonstrate the method's effectiveness, though no specific numerical results are provided in the abstract.
Differentiable architecture search (DARTS) is a promising end to end NAS method which directly optimizes the architecture parameters through general gradient descent. However, DARTS is brittle to the catastrophic failure incurred by the skip connection in the search space. Recent studies also cast doubt on the basic underlying hypotheses of DARTS which are argued to be inherently prone to the performance discrepancy between the continuous-relaxed supernet in the training phase and the discretized finalnet in the evaluation phase. We figure out that the robustness problem and the skepticism can both be explained by the information bypass leakage during the training of the supernet. This naturally highlights the vital role of the sparsity of architecture parameters in the training phase which has not been well developed in the past. We thus propose a novel sparse-regularized approximation and an efficient mixed-sparsity training scheme to robustify DARTS by eliminating the information bypass leakage. We subsequently conduct extensive experiments on multiple search spaces to demonstrate the effectiveness of our method.