Efficient Domain-Adaptive Multi-Task Dense Prediction with Vision Foundation Models
This addresses domain adaptation for multi-task learning in robotics, offering an incremental improvement over existing methods by leveraging foundation models for efficiency.
The paper tackles the problem of domain shift in multi-task dense prediction for robotics by introducing FAMDA, a framework that uses Vision Foundation Models as teachers in a self-training paradigm to generate pseudo-labels, achieving state-of-the-art performance on benchmarks and enabling models over 10x smaller than foundation models with SOTA accuracy.
Multi-task dense prediction, which aims to jointly solve tasks like semantic segmentation and depth estimation, is crucial for robotics applications but suffers from domain shift when deploying models in new environments. While unsupervised domain adaptation (UDA) addresses this challenge for single tasks, existing multi-task UDA methods primarily rely on adversarial learning approaches that are less effective than recent self-training techniques. In this paper, we introduce FAMDA, a simple yet effective UDA framework that bridges this gap by leveraging Vision Foundation Models (VFMs) as powerful teachers. Our approach integrates Segmentation and Depth foundation models into a self-training paradigm to generate high-quality pseudo-labels for the target domain, effectively distilling their robust generalization capabilities into a single, efficient student network. Extensive experiments show that FAMDA achieves state-of-the-art (SOTA) performance on standard synthetic-to-real UDA multi-task learning (MTL) benchmarks and a challenging new day-to-night adaptation task. Our framework enables the training of highly efficient models; a lightweight variant achieves SOTA accuracy while being more than 10$\times$ smaller than foundation models, highlighting FAMDA's suitability for creating domain-adaptive and efficient models for resource-constrained robotics applications.