A Score Filter Enhanced Data Assimilation Framework for Data-Driven Dynamical Systems
This work addresses the problem of long-term performance deterioration in ML models for dynamical systems, which is incremental as it combines existing data assimilation and ML techniques.
The paper tackles predictive uncertainty in machine learning models for dynamical system forecasting by integrating a score-filter-enhanced data assimilation framework, resulting in reduced uncertainty in predictions for systems like Lorenz-96 and KdV equations.
We introduce a score-filter-enhanced data assimilation framework designed to reduce predictive uncertainty in machine learning (ML) models for data-driven dynamical system forecasting. Machine learning serves as an efficient numerical model for predicting dynamical systems. However, even with sufficient data, model uncertainty remains and accumulates over time, causing the long-term performance of ML models to deteriorate. To overcome this difficulty, we integrate data assimilation techniques into the training process to iteratively refine the model predictions by incorporating observational information. Specifically, we apply the Ensemble Score Filter (EnSF), a generative AI-based training-free diffusion model approach, for solving the data assimilation problem in high-dimensional nonlinear complex systems. This leads to a hybrid data assimilation-training framework that combines ML with EnSF to improve long-term predictive performance. We shall demonstrate that EnSF-enhanced ML can effectively reduce predictive uncertainty in ML-based Lorenz-96 system prediction and the Korteweg-De Vries (KdV) equation prediction.