Learning Optical Flow, Depth, and Scene Flow without Real-World Labels
This addresses a key challenge for autonomous driving and robotics by enabling scalable 3D perception without real-world labels, though it is incremental as it builds on existing architectures like RAFT.
The paper tackles the ill-posed problem of self-supervised joint learning of depth, optical flow, and scene flow in dynamic scenes by proposing DRAFT, which combines synthetic data with geometric self-supervision and achieves new state-of-the-art results on the KITTI benchmark.
Self-supervised monocular depth estimation enables robots to learn 3D perception from raw video streams. This scalable approach leverages projective geometry and ego-motion to learn via view synthesis, assuming the world is mostly static. Dynamic scenes, which are common in autonomous driving and human-robot interaction, violate this assumption. Therefore, they require modeling dynamic objects explicitly, for instance via estimating pixel-wise 3D motion, i.e. scene flow. However, the simultaneous self-supervised learning of depth and scene flow is ill-posed, as there are infinitely many combinations that result in the same 3D point. In this paper we propose DRAFT, a new method capable of jointly learning depth, optical flow, and scene flow by combining synthetic data with geometric self-supervision. Building upon the RAFT architecture, we learn optical flow as an intermediate task to bootstrap depth and scene flow learning via triangulation. Our algorithm also leverages temporal and geometric consistency losses across tasks to improve multi-task learning. Our DRAFT architecture simultaneously establishes a new state of the art in all three tasks in the self-supervised monocular setting on the standard KITTI benchmark. Project page: https://sites.google.com/tri.global/draft.