Deep Latent State Space Models for Time-Series Generation
This work addresses the challenge of modeling highly stochastic time-series data with sharp transitions for applications in forecasting and sequence generation, representing an incremental improvement over existing methods.
The paper tackled the problem of generating time-series data with sharp transitions, proposing LS4, a generative model that outperforms previous continuous-time models on real-world datasets, achieving significant improvements in mean squared error and variational lower bounds while being 100 times faster on long sequences.
Methods based on ordinary differential equations (ODEs) are widely used to build generative models of time-series. In addition to high computational overhead due to explicitly computing hidden states recurrence, existing ODE-based models fall short in learning sequence data with sharp transitions - common in many real-world systems - due to numerical challenges during optimization. In this work, we propose LS4, a generative model for sequences with latent variables evolving according to a state space ODE to increase modeling capacity. Inspired by recent deep state space models (S4), we achieve speedups by leveraging a convolutional representation of LS4 which bypasses the explicit evaluation of hidden states. We show that LS4 significantly outperforms previous continuous-time generative models in terms of marginal distribution, classification, and prediction scores on real-world datasets in the Monash Forecasting Repository, and is capable of modeling highly stochastic data with sharp temporal transitions. LS4 sets state-of-the-art for continuous-time latent generative models, with significant improvement of mean squared error and tighter variational lower bounds on irregularly-sampled datasets, while also being x100 faster than other baselines on long sequences.