Amortised Inference in Neural Networks for Small-Scale Probabilistic Meta-Learning
This work addresses the problem of scalable and efficient probabilistic meta-learning for researchers and practitioners in machine learning, but it appears incremental as it builds on existing variational approximations and amortised inference techniques.
The paper tackles the challenge of performing Bayesian inference for task-specific Bayesian neural networks (BNNs) by proposing an amortised inference approach that replaces inducing inputs with actual data, enabling meta-learning across related datasets. The result is a method that efficiently learns inference networks to approximate posterior distributions, though no concrete performance numbers are provided in the abstract.
The global inducing point variational approximation for BNNs is based on using a set of inducing inputs to construct a series of conditional distributions that accurately approximate the conditionals of the true posterior distribution. Our key insight is that these inducing inputs can be replaced by the actual data, such that the variational distribution consists of a set of approximate likelihoods for each datapoint. This structure lends itself to amortised inference, in which the parameters of each approximate likelihood are obtained by passing each datapoint through a meta-model known as the inference network. By training this inference network across related datasets, we can meta-learn Bayesian inference over task-specific BNNs.