Stacking and stability
This provides theoretical insights into stacking for researchers and practitioners in machine learning, though it is incremental as it builds on existing stability frameworks.
The paper tackles the lack of theoretical understanding of stacking by analyzing its stability, showing that stacking's hypothesis stability is a product of base models and combiner stability, and that subsampling and bootstrap sampling improve stacking stability while stacking enhances subbagging and bagging stability.
Stacking is a general approach for combining multiple models toward greater predictive accuracy. It has found various application across different domains, ensuing from its meta-learning nature. Our understanding, nevertheless, on how and why stacking works remains intuitive and lacking in theoretical insight. In this paper, we use the stability of learning algorithms as an elemental analysis framework suitable for addressing the issue. To this end, we analyze the hypothesis stability of stacking, bag-stacking, and dag-stacking and establish a connection between bag-stacking and weighted bagging. We show that the hypothesis stability of stacking is a product of the hypothesis stability of each of the base models and the combiner. Moreover, in bag-stacking and dag-stacking, the hypothesis stability depends on the sampling strategy used to generate the training set replicates. Our findings suggest that 1) subsampling and bootstrap sampling improve the stability of stacking, and 2) stacking improves the stability of both subbagging and bagging.