A Trajectory-Based Bayesian Approach to Multi-Objective Hyperparameter Optimization with Epoch-Aware Trade-Offs
This work addresses a specific bottleneck in hyperparameter optimization for machine learning practitioners, offering an incremental improvement by better leveraging iterative training data.
The paper tackles the underutilization of epoch-wise performance insights in multi-objective hyperparameter optimization by proposing an enhanced problem formulation that treats training epochs as a decision variable to account for earlier trade-offs, such as overfitting, and introduces a trajectory-based Bayesian optimization algorithm with a novel acquisition function and early stopping mechanism, demonstrating effective identification of trade-offs and improved tuning efficiency in experiments on synthetic simulations and benchmarks.
Training machine learning models inherently involves a resource-intensive and noisy iterative learning procedure that allows epoch-wise monitoring of the model performance. However, the insights gained from the iterative learning procedure typically remain underutilized in multi-objective hyperparameter optimization scenarios. Despite the limited research in this area, existing methods commonly identify the trade-offs only at the end of model training, overlooking the fact that trade-offs can emerge at earlier epochs in cases such as overfitting. To bridge this gap, we propose an enhanced multi-objective hyperparameter optimization problem that treats the number of training epochs as a decision variable, rather than merely an auxiliary parameter, to account for trade-offs at an earlier training stage. To solve this problem and accommodate its iterative learning, we then present a trajectory-based multi-objective Bayesian optimization algorithm characterized by two features: 1) a novel acquisition function that captures the improvement along the predictive trajectory of model performances over epochs for any hyperparameter setting and 2) a multi-objective early stopping mechanism that determines when to terminate the training to maximize epoch efficiency. Experiments on synthetic simulations and hyperparameter tuning benchmarks demonstrate that our algorithm can effectively identify the desirable trade-offs while improving tuning efficiency.