Multi-objective Asynchronous Successive Halving
This work addresses the need for scalable multi-objective hyperparameter optimization in real-world applications where accuracy alone is insufficient, though it is incremental as it builds on existing asynchronous successive halving methods.
The paper tackles multi-objective hyperparameter optimization by extending asynchronous successive halving to handle multiple conflicting metrics, showing that considering the entire Pareto front outperforms scalarization methods in wall-clock time across tasks like neural architecture search and algorithmic fairness.
Hyperparameter optimization (HPO) is increasingly used to automatically tune the predictive performance (e.g., accuracy) of machine learning models. However, in a plethora of real-world applications, accuracy is only one of the multiple -- often conflicting -- performance criteria, necessitating the adoption of a multi-objective (MO) perspective. While the literature on MO optimization is rich, few prior studies have focused on HPO. In this paper, we propose algorithms that extend asynchronous successive halving (ASHA) to the MO setting. Considering multiple evaluation metrics, we assess the performance of these methods on three real world tasks: (i) Neural architecture search, (ii) algorithmic fairness and (iii) language model optimization. Our empirical analysis shows that MO ASHA enables to perform MO HPO at scale. Further, we observe that that taking the entire Pareto front into account for candidate selection consistently outperforms multi-fidelity HPO based on MO scalarization in terms of wall-clock time. Our algorithms (to be open-sourced) establish new baselines for future research in the area.