Unsupervised Volumetric Animation
This addresses the challenge of creating animatable 3D models from limited data for applications in computer vision and graphics, though it appears incremental as it builds on existing unsupervised and 3D reconstruction methods.
The paper tackles the problem of unsupervised 3D animation of deformable objects by learning structure and dynamics from single-view RGB videos, achieving tasks like 3D segmentation, keypoint estimation, and novel view synthesis without supervision.
We propose a novel approach for unsupervised 3D animation of non-rigid deformable objects. Our method learns the 3D structure and dynamics of objects solely from single-view RGB videos, and can decompose them into semantically meaningful parts that can be tracked and animated. Using a 3D autodecoder framework, paired with a keypoint estimator via a differentiable PnP algorithm, our model learns the underlying object geometry and parts decomposition in an entirely unsupervised manner. This allows it to perform 3D segmentation, 3D keypoint estimation, novel view synthesis, and animation. We primarily evaluate the framework on two video datasets: VoxCeleb $256^2$ and TEDXPeople $256^2$. In addition, on the Cats $256^2$ image dataset, we show it even learns compelling 3D geometry from still images. Finally, we show our model can obtain animatable 3D objects from a single or few images. Code and visual results available on our project website, see https://snap-research.github.io/unsupervised-volumetric-animation .