VersaViT: Enhancing MLLM Vision Backbones via Task-Guided Optimization
This work addresses the need for more robust vision backbones in multimodal AI systems, offering an incremental improvement by enhancing existing MLLM encoders for better dense prediction capabilities.
The paper tackled the problem of MLLM vision encoders having suboptimal dense feature representations for classic vision tasks like semantic segmentation and depth estimation, and proposed VersaViT, a multi-task framework that improved performance across various downstream tasks, yielding a versatile backbone for both language and pixel-level understanding.
Multimodal Large Language Models (MLLMs) have recently achieved remarkable success in visual-language understanding, demonstrating superior high-level semantic alignment within their vision encoders. An important question thus arises: Can these encoders serve as versatile vision backbones, capable of reliably performing classic vision-centric tasks as well? To address the question, we make the following contributions: (i) we identify that the vision encoders within MLLMs exhibit deficiencies in their dense feature representations, as evidenced by their suboptimal performance on dense prediction tasks (e.g., semantic segmentation, depth estimation); (ii) we propose VersaViT, a well-rounded vision transformer that instantiates a novel multi-task framework for collaborative post-training. This framework facilitates the optimization of the vision backbone via lightweight task heads with multi-granularity supervision; (iii) extensive experiments across various downstream tasks demonstrate the effectiveness of our method, yielding a versatile vision backbone suited for both language-mediated reasoning and pixel-level understanding.