A competitive baseline for deep learning enhanced data assimilation using conditional Gaussian ensemble Kalman filtering
This work addresses data assimilation challenges in high-dimensional systems like weather forecasting, offering a competitive alternative to deep learning methods, though it is incremental as it builds on existing EnKF frameworks.
The paper tackled the problem of data assimilation under nonlinear perturbations by proposing two extensions of the Ensemble Kalman Filter (CG-EnKF and NS-EnKF), which outperformed a state-of-the-art deep learning particle filter (SF) on the Lorenz-96 system, with NS-EnKF typically achieving better results.
Ensemble Kalman Filtering (EnKF) is a popular technique for data assimilation, with far ranging applications. However, the vanilla EnKF framework is not well-defined when perturbations are nonlinear. We study two non-linear extensions of the vanilla EnKF - dubbed the conditional-Gaussian EnKF (CG-EnKF) and the normal score EnKF (NS-EnKF) - which sidestep assumptions of linearity by constructing the Kalman gain matrix with the `conditional Gaussian' update formula in place of the traditional one. We then compare these models against a state-of-the-art deep learning based particle filter called the score filter (SF). This model uses an expensive score diffusion model for estimating densities and also requires a strong assumption on the perturbation operator for validity. In our comparison, we find that CG-EnKF and NS-EnKF dramatically outperform SF for a canonical problem in high-dimensional multiscale data assimilation given by the Lorenz-96 system. Our analysis also demonstrates that the CG-EnKF and NS-EnKF can handle highly non-Gaussian additive noise perturbations, with the latter typically outperforming the former.