Dynamic Online Ensembles of Basis Expansions
This work addresses the problem of scalable and flexible Bayesian modeling for practitioners, offering incremental improvements in ensembling techniques.
The paper tackles the challenge of practical Bayesian learning requiring online inference, dynamic models, and ensembling by generalizing random feature approximations to any basis expansion model, showing that alternative expansions like Hilbert space Gaussian processes often yield better performance, and proposing a method to ensemble static and dynamic models together.
Practical Bayesian learning often requires (1) online inference, (2) dynamic models, and (3) ensembling over multiple different models. Recent advances have shown how to use random feature approximations to achieve scalable, online ensembling of Gaussian processes with desirable theoretical properties and fruitful applications. One key to these methods' success is the inclusion of a random walk on the model parameters, which makes models dynamic. We show that these methods can be generalized easily to any basis expansion model and that using alternative basis expansions, such as Hilbert space Gaussian processes, often results in better performance. To simplify the process of choosing a specific basis expansion, our method's generality also allows the ensembling of several entirely different models, for example, a Gaussian process and polynomial regression. Finally, we propose a novel method to ensemble static and dynamic models together.