Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentation
This addresses the problem of multi-task learning in computer vision for applications like robotics and autonomous driving, but it is incremental as it builds on existing propagation methods.
The paper tackles the joint prediction of depth, surface normal, and semantic segmentation by proposing a Pattern-Affinitive Propagation framework that uses cross-task and task-specific propagations to diffuse similar patterns, achieving state-of-the-art or competitive results on NYUD-v2, SUN-RGBD, and KITTI datasets.
In this paper, we propose a novel Pattern-Affinitive Propagation (PAP) framework to jointly predict depth, surface normal and semantic segmentation. The motivation behind it comes from the statistic observation that pattern-affinitive pairs recur much frequently across different tasks as well as within a task. Thus, we can conduct two types of propagations, cross-task propagation and task-specific propagation, to adaptively diffuse those similar patterns. The former integrates cross-task affinity patterns to adapt to each task therein through the calculation on non-local relationships. Next the latter performs an iterative diffusion in the feature space so that the cross-task affinity patterns can be widely-spread within the task. Accordingly, the learning of each task can be regularized and boosted by the complementary task-level affinities. Extensive experiments demonstrate the effectiveness and the superiority of our method on the joint three tasks. Meanwhile, we achieve the state-of-the-art or competitive results on the three related datasets, NYUD-v2, SUN-RGBD and KITTI.