Self-Supervised Equivariant Scene Synthesis from Video
This addresses the need for unsupervised extraction and synthesis of interpretable scene elements in video data, which is incremental as it builds on prior work in scene representation learning.
The paper tackles the problem of learning scene representations from video by automatically delineating background, characters, and animations in a self-supervised manner, achieving real-time manipulation to create unseen combinations of these components.
We propose a self-supervised framework to learn scene representations from video that are automatically delineated into background, characters, and their animations. Our method capitalizes on moving characters being equivariant with respect to their transformation across frames and the background being constant with respect to that same transformation. After training, we can manipulate image encodings in real time to create unseen combinations of the delineated components. As far as we know, we are the first method to perform unsupervised extraction and synthesis of interpretable background, character, and animation. We demonstrate results on three datasets: Moving MNIST with backgrounds, 2D video game sprites, and Fashion Modeling.