MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization
This work addresses the need for efficient multi-objective optimization in real-world ML applications, offering a flexible solution for tasks like accuracy, latency, and fairness, though it is incremental as it extends an existing method.
The paper tackles the challenge of multi-objective hyperparameter optimization by proposing MO-DEHB, an evolutionary-based Hyperband method, which achieved the best performance across 15 diverse benchmarks including HPO, NAS, and joint tasks.
Hyperparameter optimization (HPO) is a powerful technique for automating the tuning of machine learning (ML) models. However, in many real-world applications, accuracy is only one of multiple performance criteria that must be considered. Optimizing these objectives simultaneously on a complex and diverse search space remains a challenging task. In this paper, we propose MO-DEHB, an effective and flexible multi-objective (MO) optimizer that extends the recent evolutionary Hyperband method DEHB. We validate the performance of MO-DEHB using a comprehensive suite of 15 benchmarks consisting of diverse and challenging MO problems, including HPO, neural architecture search (NAS), and joint NAS and HPO, with objectives including accuracy, latency and algorithmic fairness. A comparative study against state-of-the-art MO optimizers demonstrates that MO-DEHB clearly achieves the best performance across our 15 benchmarks.