Half Search Space is All You Need
This incremental improvement addresses memory efficiency for researchers and practitioners using NAS, particularly in resource-constrained settings.
The paper tackles the high GPU memory requirements in One-Shot Neural Architecture Search (NAS) by proposing an automatic pruning method that uses Zero-Shot NAS to remove low-performing architectures before applying One-Shot NAS, reducing memory consumption by 81% while maintaining accuracy.
Neural Architecture Search (NAS) is a powerful tool for automating architecture design. One-Shot NAS techniques, such as DARTS, have gained substantial popularity due to their combination of search efficiency with simplicity of implementation. By design, One-Shot methods have high GPU memory requirements during the search. To mitigate this issue, we propose to prune the search space in an efficient automatic manner to reduce memory consumption and search time while preserving the search accuracy. Specifically, we utilise Zero-Shot NAS to efficiently remove low-performing architectures from the search space before applying One-Shot NAS to the pruned search space. Experimental results on the DARTS search space show that our approach reduces memory consumption by 81% compared to the baseline One-Shot setup while achieving the same level of accuracy.