Reducing The Search Space For Hyperparameter Optimization Using Group Sparsity
This work addresses hyperparameter selection for machine learning practitioners, but it is incremental as it builds on existing spectral and bandit-based methods.
The paper tackled hyperparameter optimization by proposing a new algorithm that modifies Harmonica with group-sparse recovery and integrates it with HyperBand, leading to improvements over methods like Successive Halving and Random Search, as confirmed by experiments on CIFAR-10.
We propose a new algorithm for hyperparameter selection in machine learning algorithms. The algorithm is a novel modification of Harmonica, a spectral hyperparameter selection approach using sparse recovery methods. In particular, we show that a special encoding of hyperparameter space enables a natural group-sparse recovery formulation, which when coupled with HyperBand (a multi-armed bandit strategy) leads to improvement over existing hyperparameter optimization methods such as Successive Halving and Random Search. Experimental results on image datasets such as CIFAR-10 confirm the benefits of our approach.