Multi-objective Hyperparameter Optimization in the Age of Deep Learning
This addresses a blind spot for deep learning practitioners who need to optimize multiple objectives, though it is incremental as it builds on existing HPO methods.
The paper tackles the lack of hyperparameter optimization algorithms that incorporate prior knowledge for multiple objectives in deep learning, introducing PriMO, which achieves state-of-the-art performance across 8 benchmarks.
While Deep Learning (DL) experts often have prior knowledge about which hyperparameter settings yield strong performance, only few Hyperparameter Optimization (HPO) algorithms can leverage such prior knowledge and none incorporate priors over multiple objectives. As DL practitioners often need to optimize not just one but many objectives, this is a blind spot in the algorithmic landscape of HPO. To address this shortcoming, we introduce PriMO, the first HPO algorithm that can integrate multi-objective user beliefs. We show PriMO achieves state-of-the-art performance across 8 DL benchmarks in the multi-objective and single-objective setting, clearly positioning itself as the new go-to HPO algorithm for DL practitioners.