Parallel Architecture and Hyperparameter Search via Successive Halving and Classification
This work addresses the need for efficient parallel optimization methods in machine learning, offering a tunable-free solution for tasks like hyperparameter and architecture search, though it is incremental as it builds on existing successive halving techniques.
The paper tackles the problem of parallel black box optimization by introducing Successive Halving and Classification (SHAC), a simple algorithm that uses binary classifiers to cull undesirable search space regions in stages, achieving competitive performance in optimizing synthetic functions, hyperparameters, and architectures.
We present a simple and powerful algorithm for parallel black box optimization called Successive Halving and Classification (SHAC). The algorithm operates in $K$ stages of parallel function evaluations and trains a cascade of binary classifiers to iteratively cull the undesirable regions of the search space. SHAC is easy to implement, requires no tuning of its own configuration parameters, is invariant to the scale of the objective function and can be built using any choice of binary classifier. We adopt tree-based classifiers within SHAC and achieve competitive performance against several strong baselines for optimizing synthetic functions, hyperparameters and architectures.