Interpretable Representation Learning from Videos using Nonlinear Priors
This work addresses the problem of making machine decisions understandable and improving generalization for researchers in AI and computer vision, though it is incremental as it builds on existing VAE methods.
The paper tackles the challenge of learning interpretable representations from videos by proposing a deep learning framework that incorporates nonlinear priors, such as Newtonian physics, to enable generation of physically correct videos for hypothetical scenarios not seen during training, validated on real-world physics videos like pendulums and falling objects.
Learning interpretable representations of visual data is an important challenge, to make machines' decisions understandable to humans and to improve generalisation outside of the training distribution. To this end, we propose a deep learning framework where one can specify nonlinear priors for videos (e.g. of Newtonian physics) that allow the model to learn interpretable latent variables and use these to generate videos of hypothetical scenarios not observed at training time. We do this by extending the Variational Auto-Encoder (VAE) prior from a simple isotropic Gaussian to an arbitrary nonlinear temporal Additive Noise Model (ANM), which can describe a large number of processes (e.g. Newtonian physics). We propose a novel linearization method that constructs a Gaussian Mixture Model (GMM) approximating the prior, and derive a numerically stable Monte Carlo estimate of the KL divergence between the posterior and prior GMMs. We validate the method on different real-world physics videos including a pendulum, a mass on a spring, a falling object and a pulsar (rotating neutron star). We specify a physical prior for each experiment and show that the correct variables are learned. Once a model is trained, we intervene on it to change different physical variables (such as oscillation amplitude or adding air drag) to generate physically correct videos of hypothetical scenarios that were not observed previously.