Alternators For Sequence Modeling
This provides a versatile tool for researchers in fields like neuroscience and climate science to model and forecast sequential data, though it appears incremental as it builds on existing dynamical modeling approaches.
The paper tackles the problem of modeling complex sequential data by introducing alternators, a non-Markovian dynamical model with two neural networks that alternate between observation and feature spaces, and it shows that alternators outperform strong baselines like Mambas, neural ODEs, and diffusion models in applications such as chaotic systems, neuroscience, and climate science.
This paper introduces alternators, a novel family of non-Markovian dynamical models for sequences. An alternator features two neural networks: the observation trajectory network (OTN) and the feature trajectory network (FTN). The OTN and the FTN work in conjunction, alternating between outputting samples in the observation space and some feature space, respectively, over a cycle. The parameters of the OTN and the FTN are not time-dependent and are learned via a minimum cross-entropy criterion over the trajectories. Alternators are versatile. They can be used as dynamical latent-variable generative models or as sequence-to-sequence predictors. Alternators can uncover the latent dynamics underlying complex sequential data, accurately forecast and impute missing data, and sample new trajectories. We showcase the capabilities of alternators in three applications. We first used alternators to model the Lorenz equations, often used to describe chaotic behavior. We then applied alternators to Neuroscience, to map brain activity to physical activity. Finally, we applied alternators to Climate Science, focusing on sea-surface temperature forecasting. In all our experiments, we found alternators are stable to train, fast to sample from, yield high-quality generated samples and latent variables, and often outperform strong baselines such as Mambas, neural ODEs, and diffusion models in the domains we studied.