Minimax Optimal Bayesian Aggregation
This work addresses the challenge of robust and efficient model aggregation for regression tasks, offering a theoretically grounded method that is incremental but with strong guarantees.
The paper tackles the problem of improving prediction accuracy through ensemble methods by proposing Bayesian convex and linear aggregation approaches, showing they achieve minimax optimality when the true model is a combination of listed models and adapt to sparsity without tuning parameters.
It is generally believed that ensemble approaches, which combine multiple algorithms or models, can outperform any single algorithm at machine learning tasks, such as prediction. In this paper, we propose Bayesian convex and linear aggregation approaches motivated by regression applications. We show that the proposed approach is minimax optimal when the true data-generating model is a convex or linear combination of models in the list. Moreover, the method can adapt to sparsity structure in which certain models should receive zero weights, and the method is tuning parameter free unlike competitors. More generally, under an M-open view when the truth falls outside the space of all convex/linear combinations, our theory suggests that the posterior measure tends to concentrate on the best approximation of the truth at the minimax rate. We illustrate the method through simulation studies and several applications.