Hyperparameter Loss Surfaces Are Simple Near their Optima
This work addresses the challenge of hyperparameter tuning for large models by providing foundational tools for researchers and practitioners, though it is incremental in building on existing random search methods.
The paper tackles the problem of understanding hyperparameter loss surfaces, which are complex but become simple near optima, characterized by features like effective dimension and best possible loss. The result is a new theory and tools, including a novel technique based on random search that yields a distribution for best scores, enabling analyses such as confidence intervals for performance and effective hyperparameter counts.
Hyperparameters greatly impact models' capabilities; however, modern models are too large for extensive search. Instead, researchers design recipes that train well across scales based on their understanding of the hyperparameters. Despite this importance, few tools exist for understanding the hyperparameter loss surface. We discover novel structure in it and propose a new theory yielding such tools. The loss surface is complex, but as you approach the optimum simple structure emerges. It becomes characterized by a few basic features, like its effective dimension and the best possible loss. To uncover this asymptotic regime, we develop a novel technique based on random search. Within this regime, the best scores from random search take on a new distribution we discover. Its parameters are exactly the features defining the loss surface in the asymptotic regime. From these features, we derive a new asymptotic law for random search that can explain and extrapolate its convergence. These new tools enable new analyses, such as confidence intervals for the best possible performance or determining the effective number of hyperparameters. We make these tools available at https://github.com/nicholaslourie/opda .