On numerical approximation schemes for expectation propagation
This work addresses computational efficiency and stability issues in expectation propagation for machine learning practitioners, but it is incremental as it compares existing approximation methods rather than introducing a new paradigm.
The paper studied numerical approximation strategies for expectation propagation in large-scale learning, finding that variational sampling achieved the best convergence for training linear binary classifiers, while Laplace methods worked well with smooth factors but were unstable with non-differentiable ones, and Gaussian quadrature often led to instability or sub-optimal solutions.
Several numerical approximation strategies for the expectation-propagation algorithm are studied in the context of large-scale learning: the Laplace method, a faster variant of it, Gaussian quadrature, and a deterministic version of variational sampling (i.e., combining quadrature with variational approximation). Experiments in training linear binary classifiers show that the expectation-propagation algorithm converges best using variational sampling, while it also converges well using Laplace-style methods with smooth factors but tends to be unstable with non-differentiable ones. Gaussian quadrature yields unstable behavior or convergence to a sub-optimal solution in most experiments.