Implicit Maximum a Posteriori Filtering via Adaptive Optimization
This provides a scalable solution for high-dimensional filtering problems, such as in neural networks, though it is incremental as it builds on existing optimization techniques.
The paper tackles the computational impracticality of Bayesian filtering in high-dimensional state spaces, such as neural network weights, by reframing it as an optimization problem using an implicit optimizer, resulting in filters that are effective, robust, and scalable, with experiments showing competitive performance against standard methods.
Bayesian filtering approximates the true underlying behavior of a time-varying system by inverting an explicit generative model to convert noisy measurements into state estimates. This process typically requires either storage, inversion, and multiplication of large matrices or Monte Carlo estimation, neither of which are practical in high-dimensional state spaces such as the weight spaces of artificial neural networks. Here, we frame the standard Bayesian filtering problem as optimization over a time-varying objective. Instead of maintaining matrices for the filtering equations or simulating particles, we specify an optimizer that defines the Bayesian filter implicitly. In the linear-Gaussian setting, we show that every Kalman filter has an equivalent formulation using K steps of gradient descent. In the nonlinear setting, our experiments demonstrate that our framework results in filters that are effective, robust, and scalable to high-dimensional systems, comparing well against the standard toolbox of Bayesian filtering solutions. We suggest that it is easier to fine-tune an optimizer than it is to specify the correct filtering equations, making our framework an attractive option for high-dimensional filtering problems.