Likelihood approximations via Gaussian approximate inference
This work addresses computational bottlenecks for researchers and practitioners in machine learning dealing with non-Gaussian likelihoods, offering incremental improvements in approximation efficiency.
The paper tackles the computational challenges of non-Gaussian likelihoods in modeling complex observations by proposing Gaussian approximation schemes based on variational inference and moment matching, which achieve good approximation quality in classification tasks and outperform existing methods in streaming problems.
Non-Gaussian likelihoods are essential for modelling complex real-world observations but pose significant computational challenges in learning and inference. Even with Gaussian priors, non-Gaussian likelihoods often lead to analytically intractable posteriors, necessitating approximation methods. To this end, we propose efficient schemes to approximate the effects of non-Gaussian likelihoods by Gaussian densities based on variational inference and moment matching in transformed bases. These enable efficient inference strategies originally designed for models with a Gaussian likelihood to be deployed. Our empirical results demonstrate that the proposed matching strategies attain good approximation quality for binary and multiclass classification in large-scale point-estimate and distributional inferential settings. In challenging streaming problems, the proposed methods outperform all existing likelihood approximations and approximate inference methods in the exact models. As a by-product, we show that the proposed approximate log-likelihoods are a superior alternative to least-squares on raw labels for neural network classification.