Bayesian hierarchical stacking: Some models are (somewhere) useful
This work addresses the need for more flexible and effective model averaging in statistical modeling, particularly for scenarios with heterogeneous predictive performance, though it appears incremental as an extension of existing stacking methods.
The authors tackled the problem of improving model averaging when predictive performance varies across inputs by introducing Bayesian hierarchical stacking, which allows model weights to vary as a function of data and incorporates structured priors. They demonstrated performance gains through theoretical bounds and applied examples.
Stacking is a widely used model averaging technique that asymptotically yields optimal predictions among linear averages. We show that stacking is most effective when model predictive performance is heterogeneous in inputs, and we can further improve the stacked mixture with a hierarchical model. We generalize stacking to Bayesian hierarchical stacking. The model weights are varying as a function of data, partially-pooled, and inferred using Bayesian inference. We further incorporate discrete and continuous inputs, other structured priors, and time series and longitudinal data. To verify the performance gain of the proposed method, we derive theory bounds, and demonstrate on several applied problems.