Creatures great and SMAL: Recovering the shape and motion of animals from video
This addresses the challenge of 3D animal pose estimation from video, which is incremental as it builds on existing methods but adapts to limited training data by using silhouettes.
The paper tackles the problem of recovering 3D shape and motion of quadrupeds from video by using a system that predicts 2D joint positions, optimizes correspondences, and fits a 3D model, achieving accurate reconstructions across multiple species.
We present a system to recover the 3D shape and motion of a wide variety of quadrupeds from video. The system comprises a machine learning front-end which predicts candidate 2D joint positions, a discrete optimization which finds kinematically plausible joint correspondences, and an energy minimization stage which fits a detailed 3D model to the image. In order to overcome the limited availability of motion capture training data from animals, and the difficulty of generating realistic synthetic training images, the system is designed to work on silhouette data. The joint candidate predictor is trained on synthetically generated silhouette images, and at test time, deep learning methods or standard video segmentation tools are used to extract silhouettes from real data. The system is tested on animal videos from several species, and shows accurate reconstructions of 3D shape and pose.