Self-Organizing Score-based Data Assimilation
This work addresses a specific bottleneck in state-space modeling for domains like neuroscience and atmospheric science, offering an incremental improvement by extending SDA to handle unknown parameters.
The paper tackles the challenge of joint inference of latent states and unknown parameters in nonlinear, high-dimensional state-space models by proposing a framework that integrates self-organization techniques into Score-based Data Assimilation (SDA), enabling scalable joint inference with validation on high-dimensional systems up to several hundred thousand dimensions.
A state-space model is a statistical framework for inferring latent states from observed time-series data. However, inference with nonlinear and high-dimensional state-space models remains challenging. To this end, an approach based on diffusion models-a powerful class of deep generative models-has been developed, known as Score-based Data Assimilation (SDA). However, SDA cannot be directly applied when the latent-state transition depends on unknown parameters that must be inferred jointly with the latent states. To overcome this limitation, we propose a framework that enables SDA to handle latent states with unknown parameters. A key feature of the proposed method is the incorporation of the self-organization technique, which has been used in classical state-space modeling for the joint estimation of latent states and parameters. By integrating this classical technique into modern SDA, our method enables joint inference of latent states and unknown parameters while maintaining the high training efficiency of SDA. The effectiveness of the proposed approach is validated through numerical experiments on dynamical systems arising in neuroscience and atmospheric science. In addition, its scalability is demonstrated using a high-dimensional Kolmogorov flow, with the data dimension on the order of several hundred thousand.