A resource-efficient method for repeated HPO and NAS problems
This work addresses the computational inefficiency of repeated HNAS for researchers and practitioners, though it is incremental as it builds on existing Successive Halving methods.
The paper tackles the problem of repeated hyperparameter and neural architecture search (HNAS) by proposing an extension of Successive Halving that leverages information from previous searches to save computational resources, demonstrating drastic cost reductions while maintaining accuracy and robustness to negative transfer.
In this work we consider the problem of repeated hyperparameter and neural architecture search (HNAS). We propose an extension of Successive Halving that is able to leverage information gained in previous HNAS problems with the goal of saving computational resources. We empirically demonstrate that our solution is able to drastically decrease costs while maintaining accuracy and being robust to negative transfer. Our method is significantly simpler than competing transfer learning approaches, setting a new baseline for transfer learning in HNAS.