Distilled Semantics for Comprehensive Scene Understanding from Videos
This work addresses the problem of comprehensive scene understanding for autonomous systems, offering an incremental improvement by integrating semantics with existing unsupervised methods.
The paper tackles holistic scene understanding from monocular videos by jointly learning depth, motion, and semantics using a novel training protocol and compact architecture, achieving state-of-the-art results in monocular depth estimation, optical flow, and motion segmentation.
Whole understanding of the surroundings is paramount to autonomous systems. Recent works have shown that deep neural networks can learn geometry (depth) and motion (optical flow) from a monocular video without any explicit supervision from ground truth annotations, particularly hard to source for these two tasks. In this paper, we take an additional step toward holistic scene understanding with monocular cameras by learning depth and motion alongside with semantics, with supervision for the latter provided by a pre-trained network distilling proxy ground truth images. We address the three tasks jointly by a) a novel training protocol based on knowledge distillation and self-supervision and b) a compact network architecture which enables efficient scene understanding on both power hungry GPUs and low-power embedded platforms. We thoroughly assess the performance of our framework and show that it yields state-of-the-art results for monocular depth estimation, optical flow and motion segmentation.