Genealogical Population-Based Training for Hyperparameter Optimization
This addresses the problem of inefficient hyperparameter optimization for machine learning practitioners by introducing a novel method that breaks from the paradigm of treating hyperparameters independently of models, offering significant computational and performance gains.
The paper tackles hyperparameter optimization by proposing Genealogical Population-Based Training (GPBT), which exploits the coupling between hyperparameters and models through shared genealogical histories, resulting in a 2-3 times reduction in computational cost, a 1% accuracy improvement on computer vision tasks, and an order of magnitude reduction in variance compared to current algorithms.
HyperParameter Optimization (HPO) aims at finding the best HyperParameters (HPs) of learning models, such as neural networks, in the fastest and most efficient way possible. Most recent HPO algorithms try to optimize HPs regardless of the model that obtained them, assuming that for different models, same HPs will produce very similar results. We break free from this paradigm and propose a new take on preexisting methods that we called Genealogical Population Based Training (GPBT). GPBT, via the shared histories of "genealogically"-related models, exploit the coupling of HPs and models in an efficient way. We experimentally demonstrate that our method cuts down by 2 to 3 times the computational cost required, generally allows a 1% accuracy improvement on computer vision tasks, and reduces the variance of the results by an order of magnitude, compared to the current algorithms. Our method is search-algorithm agnostic so that the inner search routine can be any search algorithm like TPE, GP, CMA or random search.