Disentangling Multiple Features in Video Sequences using Gaussian Processes in Variational Autoencoders
This work addresses the problem of disentangling multiple features in video data for researchers in machine learning and computer vision, representing an incremental improvement over previous methods.
The paper tackles unsupervised learning of disentangled representations in video sequences by introducing MGP-VAE, which uses Gaussian processes to model the latent space and a novel geodesic loss function, resulting in improved performance on video prediction tasks compared to baselines.
We introduce MGP-VAE (Multi-disentangled-features Gaussian Processes Variational AutoEncoder), a variational autoencoder which uses Gaussian processes (GP) to model the latent space for the unsupervised learning of disentangled representations in video sequences. We improve upon previous work by establishing a framework by which multiple features, static or dynamic, can be disentangled. Specifically we use fractional Brownian motions (fBM) and Brownian bridges (BB) to enforce an inter-frame correlation structure in each independent channel, and show that varying this structure enables one to capture different factors of variation in the data. We demonstrate the quality of our representations with experiments on three publicly available datasets, and also quantify the improvement using a video prediction task. Moreover, we introduce a novel geodesic loss function which takes into account the curvature of the data manifold to improve learning. Our experiments show that the combination of the improved representations with the novel loss function enable MGP-VAE to outperform the baselines in video prediction.