3D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understanding
This work addresses the problem of unstructured features in multi-task learning for dense scene understanding, offering a domain-specific improvement for computer vision applications.
The paper tackles the challenge of training a single network for multiple dense prediction tasks like segmentation and depth estimation by integrating 3D-aware cross-view correlations to improve geometric consistency. Results on NYUv2 and PASCAL-Context datasets show that this approach effectively enhances performance in multi-task learning.
This paper addresses the challenge of training a single network to jointly perform multiple dense prediction tasks, such as segmentation and depth estimation, i.e., multi-task learning (MTL). Current approaches mainly capture cross-task relations in the 2D image space, often leading to unstructured features lacking 3D-awareness. We argue that 3D-awareness is vital for modeling cross-task correlations essential for comprehensive scene understanding. We propose to address this problem by integrating correlations across views, i.e., cost volume, as geometric consistency in the MTL network. Specifically, we introduce a lightweight Cross-view Module (CvM), shared across tasks, to exchange information across views and capture cross-view correlations, integrated with a feature from MTL encoder for multi-task predictions. This module is architecture-agnostic and can be applied to both single and multi-view data. Extensive results on NYUv2 and PASCAL-Context demonstrate that our method effectively injects geometric consistency into existing MTL methods to improve performance.