Bonsai-Net: One-Shot Neural Architecture Search via Differentiable Pruners
This addresses the computational expense of NAS for researchers and practitioners, offering an incremental improvement by relaxing the search space and integrating search with training.
The paper tackles the problem of inefficient Neural Architecture Search (NAS) by proposing Bonsai-Net, a one-shot NAS method that uses a differentiable pruner to explore a relaxed search space, resulting in state-of-the-art architectures that outperform random search with fewer parameters and reduced training time.
One-shot Neural Architecture Search (NAS) aims to minimize the computational expense of discovering state-of-the-art models. However, in the past year attention has been drawn to the comparable performance of naive random search across the same search spaces used by leading NAS algorithms. To address this, we explore the effects of drastically relaxing the NAS search space, and we present Bonsai-Net, an efficient one-shot NAS method to explore our relaxed search space. Bonsai-Net is built around a modified differential pruner and can consistently discover state-of-the-art architectures that are significantly better than random search with fewer parameters than other state-of-the-art methods. Additionally, Bonsai-Net performs simultaneous model search and training, dramatically reducing the total time it takes to generate fully-trained models from scratch.