Differential Bayesian Neural Nets
This work addresses uncertainty calibration for researchers and practitioners using continuous-time neural networks, but it is incremental as it adapts existing Bayesian and SDE techniques to N-ODEs.
The authors tackled the problem of uncertainty quantification in Neural Ordinary Differential Equations by proposing a Bayesian version that uses Bayesian Neural Nets for drift and diffusion terms in a Stochastic Differential Equation, resulting in significantly improved stability on synthetic tasks and better model fit on UCI benchmarks compared to non-Bayesian methods.
Neural Ordinary Differential Equations (N-ODEs) are a powerful building block for learning systems, which extend residual networks to a continuous-time dynamical system. We propose a Bayesian version of N-ODEs that enables well-calibrated quantification of prediction uncertainty, while maintaining the expressive power of their deterministic counterpart. We assign Bayesian Neural Nets (BNNs) to both the drift and the diffusion terms of a Stochastic Differential Equation (SDE) that models the flow of the activation map in time. We infer the posterior on the BNN weights using a straightforward adaptation of Stochastic Gradient Langevin Dynamics (SGLD). We illustrate significantly improved stability on two synthetic time series prediction tasks and report better model fit on UCI regression benchmarks with our method when compared to its non-Bayesian counterpart.