Auto-differentiable Ensemble Kalman Filters
This addresses data assimilation problems in scientific and engineering applications where state estimation is difficult due to high dimensionality and unknown dynamics, representing an incremental improvement by blending existing techniques.
The paper tackles the challenge of learning dynamical systems in data assimilation for high-dimensional states with unknown dynamics, introducing auto-differentiable ensemble Kalman filters (AD-EnKFs) that outperform existing methods like expectation-maximization or particle filters in numerical tests on the Lorenz-96 model.
Data assimilation is concerned with sequentially estimating a temporally-evolving state. This task, which arises in a wide range of scientific and engineering applications, is particularly challenging when the state is high-dimensional and the state-space dynamics are unknown. This paper introduces a machine learning framework for learning dynamical systems in data assimilation. Our auto-differentiable ensemble Kalman filters (AD-EnKFs) blend ensemble Kalman filters for state recovery with machine learning tools for learning the dynamics. In doing so, AD-EnKFs leverage the ability of ensemble Kalman filters to scale to high-dimensional states and the power of automatic differentiation to train high-dimensional surrogate models for the dynamics. Numerical results using the Lorenz-96 model show that AD-EnKFs outperform existing methods that use expectation-maximization or particle filters to merge data assimilation and machine learning. In addition, AD-EnKFs are easy to implement and require minimal tuning.