Unsupervised object-centric video generation and decomposition in 3D
This work addresses the challenge of generating and decomposing complex 3D scenes from monocular videos without supervision, which is incremental as it builds on prior 2D-based approaches by incorporating 3D scene understanding.
The paper tackles the problem of unsupervised generative modeling of videos by representing them as compositions of moving 3D objects and backgrounds, rather than 2D sprites, and shows that it outperforms state-of-the-art methods on tasks like depth prediction, 3D object detection, 2D instance segmentation, and tracking.
A natural approach to generative modeling of videos is to represent them as a composition of moving objects. Recent works model a set of 2D sprites over a slowly-varying background, but without considering the underlying 3D scene that gives rise to them. We instead propose to model a video as the view seen while moving through a scene with multiple 3D objects and a 3D background. Our model is trained from monocular videos without any supervision, yet learns to generate coherent 3D scenes containing several moving objects. We conduct detailed experiments on two datasets, going beyond the visual complexity supported by state-of-the-art generative approaches. We evaluate our method on depth-prediction and 3D object detection -- tasks which cannot be addressed by those earlier works -- and show it out-performs them even on 2D instance segmentation and tracking.