Semi-supervised Multi-task Learning for Semantics and Depth
This addresses a practical limitation in multi-task learning for computer vision applications, though it is incremental by building on existing adversarial and domain adaptation techniques.
The paper tackles the problem of multi-task learning when datasets lack full annotations for all tasks, proposing a semi-supervised method that uses adversarial learning and domain-aware discriminators to leverage partial data across datasets for semantic segmentation and depth estimation, showing effectiveness on street view and remote sensing benchmarks.
Multi-Task Learning (MTL) aims to enhance the model generalization by sharing representations between related tasks for better performance. Typical MTL methods are jointly trained with the complete multitude of ground-truths for all tasks simultaneously. However, one single dataset may not contain the annotations for each task of interest. To address this issue, we propose the Semi-supervised Multi-Task Learning (SemiMTL) method to leverage the available supervisory signals from different datasets, particularly for semantic segmentation and depth estimation tasks. To this end, we design an adversarial learning scheme in our semi-supervised training by leveraging unlabeled data to optimize all the task branches simultaneously and accomplish all tasks across datasets with partial annotations. We further present a domain-aware discriminator structure with various alignment formulations to mitigate the domain discrepancy issue among datasets. Finally, we demonstrate the effectiveness of the proposed method to learn across different datasets on challenging street view and remote sensing benchmarks.