Joint Unsupervised Learning of Optical Flow and Depth by Watching Stereo Videos
This addresses the problem of estimating scene geometry and motion without labeled data for applications like autonomous driving, though it is incremental by building on existing unsupervised methods.
The paper tackles the joint unsupervised learning of optical flow and depth from stereo videos by exploiting geometric constraints, achieving state-of-the-art results on the KITTI dataset.
Learning depth and optical flow via deep neural networks by watching videos has made significant progress recently. In this paper, we jointly solve the two tasks by exploiting the underlying geometric rules within stereo videos. Specifically, given two consecutive stereo image pairs from a video, we first estimate depth, camera ego-motion and optical flow from three neural networks. Then the whole scene is decomposed into moving foreground and static background by compar- ing the estimated optical flow and rigid flow derived from the depth and ego-motion. We propose a novel consistency loss to let the optical flow learn from the more accurate rigid flow in static regions. We also design a rigid alignment module which helps refine ego-motion estimation by using the estimated depth and optical flow. Experiments on the KITTI dataset show that our results significantly outperform other state-of- the-art algorithms. Source codes can be found at https: //github.com/baidu-research/UnDepthflow