To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing
This work addresses the efficiency of weight-sharing for researchers and practitioners in automated neural architecture design, revealing it as incremental with limited practical advantages.
The paper investigates the effectiveness of weight-sharing in Neural Architecture Search (NAS) using the NASBench dataset, finding that despite decent correlations, weight-sharing rarely provides significant benefits compared to random search, with search space characteristics influencing outcomes.
Weight-sharing (WS) has recently emerged as a paradigm to accelerate the automated search for efficient neural architectures, a process dubbed Neural Architecture Search (NAS). Although very appealing, this framework is not without drawbacks and several works have started to question its capabilities on small hand-crafted benchmarks. In this paper, we take advantage of the \nasbench dataset to challenge the efficiency of WS on a representative search space. By comparing a SOTA WS approach to a plain random search we show that, despite decent correlations between evaluations using weight-sharing and standalone ones, WS is only rarely significantly helpful to NAS. In particular we highlight the impact of the search space itself on the benefits.