SENSE: a Shared Encoder Network for Scene-flow Estimation
This work addresses scene flow estimation for computer vision applications, offering an efficient and modular solution that handles partially labeled data, though it is incremental as it builds on existing multi-task learning approaches.
The authors tackled holistic scene flow estimation by introducing SENSE, a compact network that shares encoder features across four related tasks, achieving state-of-the-art results on optical flow benchmarks with competitive performance on stereo and scene flow while using less memory.
We introduce a compact network for holistic scene flow estimation, called SENSE, which shares common encoder features among four closely-related tasks: optical flow estimation, disparity estimation from stereo, occlusion estimation, and semantic segmentation. Our key insight is that sharing features makes the network more compact, induces better feature representations, and can better exploit interactions among these tasks to handle partially labeled data. With a shared encoder, we can flexibly add decoders for different tasks during training. This modular design leads to a compact and efficient model at inference time. Exploiting the interactions among these tasks allows us to introduce distillation and self-supervised losses in addition to supervised losses, which can better handle partially labeled real-world data. SENSE achieves state-of-the-art results on several optical flow benchmarks and runs as fast as networks specifically designed for optical flow. It also compares favorably against the state of the art on stereo and scene flow, while consuming much less memory.