Parallel Multi-Objective Hyperparameter Optimization with Uniform Normalization and Bounded Objectives
This work addresses the problem of efficiently optimizing multiple performance objectives like accuracy and fairness for machine learning practitioners, representing an incremental improvement in multi-objective optimization methods.
The paper tackles the challenge of multi-objective hyperparameter optimization in machine learning by proposing a Bayesian optimization algorithm that uses uniform normalization and randomized weights, achieving a 5x speed-up with 16x more workers.
Machine learning (ML) methods offer a wide range of configurable hyperparameters that have a significant influence on their performance. While accuracy is a commonly used performance objective, in many settings, it is not sufficient. Optimizing the ML models with respect to multiple objectives such as accuracy, confidence, fairness, calibration, privacy, latency, and memory consumption is becoming crucial. To that end, hyperparameter optimization, the approach to systematically optimize the hyperparameters, which is already challenging for a single objective, is even more challenging for multiple objectives. In addition, the differences in objective scales, the failures, and the presence of outlier values in objectives make the problem even harder. We propose a multi-objective Bayesian optimization (MoBO) algorithm that addresses these problems through uniform objective normalization and randomized weights in scalarization. We increase the efficiency of our approach by imposing constraints on the objective to avoid exploring unnecessary configurations (e.g., insufficient accuracy). Finally, we leverage an approach to parallelize the MoBO which results in a 5x speed-up when using 16x more workers.