Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics
This addresses the challenge of multimodal data integration in dynamical systems reconstruction for scientific applications like neuroscience, offering a generative approach that is incremental over existing deep learning methods.
The paper tackles the problem of reconstructing nonlinear dynamical systems from multimodal time series data, such as discrete and continuous measurements, by introducing a generative framework that integrates these data sources for joint modeling, enabling reconstruction even from symbolic data alone and producing trajectories matching the ground truth system's structure.
Many, if not most, systems of interest in science are naturally described as nonlinear dynamical systems. Empirically, we commonly access these systems through time series measurements. Often such time series may consist of discrete random variables rather than continuous measurements, or may be composed of measurements from multiple data modalities observed simultaneously. For instance, in neuroscience we may have behavioral labels in addition to spike counts and continuous physiological recordings. While by now there is a burgeoning literature on deep learning for dynamical systems reconstruction (DSR), multimodal data integration has hardly been considered in this context. Here we provide such an efficient and flexible algorithmic framework that rests on a multimodal variational autoencoder for generating a sparse teacher signal that guides training of a reconstruction model, exploiting recent advances in DSR training techniques. It enables to combine various sources of information for optimal reconstruction, even allows for reconstruction from symbolic data (class labels) alone, and connects different types of observations within a common latent dynamics space. In contrast to previous multimodal data integration techniques for scientific applications, our framework is fully \textit{generative}, producing, after training, trajectories with the same geometrical and temporal structure as those of the ground truth system.