Model Agnostic Combination for Ensemble Learning
This addresses the need for flexible ensemble methods in machine learning, particularly for scenarios requiring dynamic model updates, though it is incremental as it builds on existing ensembling concepts.
The paper tackles the problem of ensemble learning by proposing MAC, a model-agnostic combination technique that allows adding or replacing sub-models without retraining, and shows it outperforms classical averaging and is competitive with XGBoost on the Kaggle RSNA Intracranial Hemorrhage Detection challenge.
Ensemble of models is well known to improve single model performance. We present a novel ensembling technique coined MAC that is designed to find the optimal function for combining models while remaining invariant to the number of sub-models involved in the combination. Being agnostic to the number of sub-models enables addition and replacement of sub-models to the combination even after deployment, unlike many of the current methods for ensembling such as stacking, boosting, mixture of experts and super learners that lock the models used for combination during training and therefore need retraining whenever a new model is introduced into the ensemble. We show that on the Kaggle RSNA Intracranial Hemorrhage Detection challenge, MAC outperforms classical average methods, demonstrates competitive results to boosting via XGBoost for a fixed number of sub-models, and outperforms it when adding sub-models to the combination without retraining.