Maximum Likelihood Learning of Latent Dynamics Without Reconstruction
This provides a method for learning task-relevant latents in time series without ad-hoc regularization, potentially benefiting applications in video analysis and other domains, though it appears incremental as it builds on existing probabilistic and contrastive approaches.
The paper tackles the problem of unsupervised learning of latent dynamics in time series data by introducing the RP-GSSM, a probabilistic model that learns Markovian Gaussian latents without needing explicit reconstruction, and it outperforms alternatives on tasks like learning nonlinear stochastic dynamics from video.
We introduce a novel unsupervised learning method for time series data with latent dynamical structure: the recognition-parametrized Gaussian state space model (RP-GSSM). The RP-GSSM is a probabilistic model that learns Markovian Gaussian latents explaining statistical dependence between observations at different time steps, combining the intuition of contrastive methods with the flexible tools of probabilistic generative models. Unlike contrastive approaches, the RP-GSSM is a valid probabilistic model learned via maximum likelihood. Unlike generative approaches, the RP-GSSM has no need for an explicit network mapping from latents to observations, allowing it to focus model capacity on inference of latents. The model is both tractable and expressive: it admits exact inference thanks to its jointly Gaussian latent prior, while maintaining expressivity with an arbitrarily nonlinear neural network link between observations and latents. These qualities allow the RP-GSSM to learn task-relevant latents without ad-hoc regularization, auxiliary losses, or optimizer scheduling. We show how this approach outperforms alternatives on problems that include learning nonlinear stochastic dynamics from video, with or without background distractors. Our results position the RP-GSSM as a useful foundation model for a variety of downstream applications.