Neural Diffusion Processes
This work addresses the need for flexible and efficient meta-learning methods for functional distributions, with applications in regression and optimization, though it appears incremental by building on existing diffusion models.
The authors tackled the problem of meta-learning distributions over functions by proposing Neural Diffusion Processes (NDPs), which learn to sample from functional distributions using denoising diffusion models and a custom attention block, resulting in performance surpassing neural processes and capturing distributions close to the true Bayesian posterior.
Neural network approaches for meta-learning distributions over functions have desirable properties such as increased flexibility and a reduced complexity of inference. Building on the successes of denoising diffusion models for generative modelling, we propose Neural Diffusion Processes (NDPs), a novel approach that learns to sample from a rich distribution over functions through its finite marginals. By introducing a custom attention block we are able to incorporate properties of stochastic processes, such as exchangeability, directly into the NDP's architecture. We empirically show that NDPs can capture functional distributions close to the true Bayesian posterior, demonstrating that they can successfully emulate the behaviour of Gaussian processes and surpass the performance of neural processes. NDPs enable a variety of downstream tasks, including regression, implicit hyperparameter marginalisation, non-Gaussian posterior prediction and global optimisation.