Subsampled Ensemble Can Improve Generalization Tail Exponentially
This work addresses the challenge of enhancing out-of-sample performance for machine learning practitioners dealing with heavy-tailed data or slow-learning models, offering a method with exponential tail improvement over traditional variance reduction.
The paper tackles the problem of improving generalization tail behavior in ensemble learning by proposing a subsampled ensemble with majority voting, achieving exponentially decaying tails for excess risk even with base learners that have slow polynomial decay rates, as demonstrated in numerical examples with heavy-tailed data.
Ensemble learning is a popular technique to improve the accuracy of machine learning models. It traditionally hinges on the rationale that aggregating multiple weak models can lead to better models with lower variance and hence higher stability, especially for discontinuous base learners. In this paper, we provide a new perspective on ensembling. By selecting the best model trained on subsamples via majority voting, we can attain exponentially decaying tails for the excess risk, even if the base learner suffers from slow (i.e., polynomial) decay rates. This tail enhancement power of ensembling is agnostic to the underlying base learner and is stronger than variance reduction in the sense of exhibiting rate improvement. We demonstrate how our ensemble methods can substantially improve out-of-sample performances in a range of numerical examples involving heavy-tailed data or intrinsically slow rates. Code for the proposed methods is available at https://github.com/mickeyhqian/VoteEnsemble.