SfM-Net: Learning of Structure and Motion from Video
This addresses the challenge of unsupervised or weakly supervised motion and depth estimation in computer vision, with incremental improvements in integrating geometry into neural networks.
The paper tackles the problem of motion estimation in videos by proposing SfM-Net, a geometry-aware neural network that decomposes pixel motion into scene depth, camera motion, and object motions, and it successfully estimates depth and camera rotations/translations while often segmenting moving objects without supervision.
We propose SfM-Net, a geometry-aware neural network for motion estimation in videos that decomposes frame-to-frame pixel motion in terms of scene and object depth, camera motion and 3D object rotations and translations. Given a sequence of frames, SfM-Net predicts depth, segmentation, camera and rigid object motions, converts those into a dense frame-to-frame motion field (optical flow), differentiably warps frames in time to match pixels and back-propagates. The model can be trained with various degrees of supervision: 1) self-supervised by the re-projection photometric error (completely unsupervised), 2) supervised by ego-motion (camera motion), or 3) supervised by depth (e.g., as provided by RGBD sensors). SfM-Net extracts meaningful depth estimates and successfully estimates frame-to-frame camera rotations and translations. It often successfully segments the moving objects in the scene, even though such supervision is never provided.