PSEO: Optimizing Post-hoc Stacking Ensemble Through Hyperparameter Tuning
This work addresses a bottleneck in AutoML for practitioners by improving ensemble performance, though it is incremental as it builds on existing ensemble and CASH methods.
The paper tackles the problem of suboptimal fixed strategies in post-hoc ensemble construction within AutoML by proposing PSEO, a framework that optimizes stacking ensembles through hyperparameter tuning, achieving the best average test rank of 2.96 among 16 methods on 80 datasets.
The Combined Algorithm Selection and Hyperparameter Optimization (CASH) problem is fundamental in Automated Machine Learning (AutoML). Inspired by the success of ensemble learning, recent AutoML systems construct post-hoc ensembles for final predictions rather than relying on the best single model. However, while most CASH methods conduct extensive searches for the optimal single model, they typically employ fixed strategies during the ensemble phase that fail to adapt to specific task characteristics. To tackle this issue, we propose PSEO, a framework for post-hoc stacking ensemble optimization. First, we conduct base model selection through binary quadratic programming, with a trade-off between diversity and performance. Furthermore, we introduce two mechanisms to fully realize the potential of multi-layer stacking. Finally, PSEO builds a hyperparameter space and searches for the optimal post-hoc ensemble strategy within it. Empirical results on 80 public datasets show that \sys achieves the best average test rank (2.96) among 16 methods, including post-hoc designs in recent AutoML systems and state-of-the-art ensemble learning methods.