Importance Weighting Approach in Kernel Bayes' Rule
This work addresses the challenge of stable and model-free Bayesian computation for researchers in machine learning, though it is incremental as it builds on existing kernel Bayes' rule methods.
The authors tackled the problem of performing Bayesian updates nonparametrically using kernel methods, introducing an importance weighting approach to the kernel Bayes' rule that avoids operator inversion for improved stability. This method achieved uniformly better empirical performance than the original kernel Bayes' rule and competitive results on synthetic benchmarks, including high-dimensional image observations.
We study a nonparametric approach to Bayesian computation via feature means, where the expectation of prior features is updated to yield expected kernel posterior features, based on regression from learned neural net or kernel features of the observations. All quantities involved in the Bayesian update are learned from observed data, making the method entirely model-free. The resulting algorithm is a novel instance of a kernel Bayes' rule (KBR), based on importance weighting. This results in superior numerical stability to the original approach to KBR, which requires operator inversion. We show the convergence of the estimator using a novel consistency analysis on the importance weighting estimator in the infinity norm. We evaluate KBR on challenging synthetic benchmarks, including a filtering problem with a state-space model involving high dimensional image observations. Importance weighted KBR yields uniformly better empirical performance than the original KBR, and competitive performance with other competing methods.