DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teachers
This work addresses the challenge of unifying diverse vision models into a single efficient encoder, which is incremental but offers practical benefits for multi-task perception systems.
The paper tackles the problem of distilling a universal encoder from heterogeneous 2D and 3D teacher models, achieving performance comparable to or better than the larger teachers on tasks like classification, segmentation, depth estimation, and 3D human perception, with notable gains such as surpassing MASt3R in Map-free Visual Relocalization.
Recent multi-teacher distillation methods have unified the encoders of multiple foundation models into a single encoder, achieving competitive performance on core vision tasks like classification, segmentation, and depth estimation. This led us to ask: Could similar success be achieved when the pool of teachers also includes vision models specialized in diverse tasks across both 2D and 3D perception? In this paper, we define and investigate the problem of heterogeneous teacher distillation, or co-distillation, a challenging multi-teacher distillation scenario where teacher models vary significantly in both (a) their design objectives and (b) the data they were trained on. We explore data-sharing strategies and teacher-specific encoding, and introduce DUNE, a single encoder excelling in 2D vision, 3D understanding, and 3D human perception. Our model achieves performance comparable to that of its larger teachers, sometimes even outperforming them, on their respective tasks. Notably, DUNE surpasses MASt3R in Map-free Visual Relocalization with a much smaller encoder.