Is Differentiable Architecture Search truly a One-Shot Method?
This work challenges the reliability of DAS for researchers in automated machine learning, highlighting incremental issues in its evaluation.
The paper investigated whether differentiable architecture search (DAS) truly functions as a one-shot method by applying it to inverse problems like signal reconstruction, showing it can extend to this domain but revealing high variance, hyperparameter sensitivity, and poor correlation between training and final performance.
Differentiable architecture search (DAS) is a widely researched tool for the discovery of novel architectures, due to its promising results for image classification. The main benefit of DAS is the effectiveness achieved through the weight-sharing one-shot paradigm, which allows efficient architecture search. In this work, we investigate DAS in a systematic case study of inverse problems, which allows us to analyze these potential benefits in a controlled manner. We demonstrate that the success of DAS can be extended from image classification to signal reconstruction, in principle. However, our experiments also expose three fundamental difficulties in the evaluation of DAS-based methods in inverse problems: First, the results show a large variance in all test cases. Second, the final performance is strongly dependent on the hyperparameters of the optimizer. And third, the performance of the weight-sharing architecture used during training does not reflect the final performance of the found architecture well. While the results on image reconstruction confirm the potential of the DAS paradigm, they challenge the common understanding of DAS as a one-shot method.