Tensor-Var: Efficient Four-Dimensional Variational Data Assimilation
This work addresses efficiency and accuracy challenges in data assimilation for complex nonlinear systems like weather prediction, offering a novel integration of deep learning with theoretical guarantees, though it is incremental as it builds on existing 4D-Var methods.
The paper tackles the high computational costs and imperfect state-observation mappings in four-dimensional variational data assimilation (4D-Var) by proposing Tensor-Var, a framework that integrates kernel conditional mean embedding to linearize nonlinear dynamics for convex optimization, resulting in improved accuracy and a 10- to 20-fold speed increase over baselines in chaotic systems and global weather prediction.
Variational data assimilation estimates the dynamical system states by minimizing a cost function that fits the numerical models with the observational data. Although four-dimensional variational assimilation (4D-Var) is widely used, it faces high computational costs in complex nonlinear systems and depends on imperfect state-observation mappings. Deep learning (DL) offers more expressive approximators, while integrating DL models into 4D-Var is challenging due to their nonlinearities and lack of theoretical guarantees in assimilation results. In this paper, we propose Tensor-Var, a novel framework that integrates kernel conditional mean embedding (CME) with 4D-Var to linearize nonlinear dynamics, achieving convex optimization in a learned feature space. Moreover, our method provides a new perspective for solving 4D-Var in a linear way, offering theoretical guarantees of consistent assimilation results between the original and feature spaces. To handle large-scale problems, we propose a method to learn deep features using neural networks within the Tensor-Var framework. Experiments on chaotic systems and global weather prediction with real-time observations show that Tensor-Var outperforms conventional and DL hybrid 4D-Var baselines in accuracy while achieving a 10- to 20-fold speed improvement.