DenseMLLM: Standard Multimodal LLMs are Intrinsic Dense Predictors
This work addresses the problem of architectural fragmentation in MLLMs for researchers and practitioners, offering a more practical and generalist approach, though it is incremental in nature.
The paper tackles the challenge of adapting standard multimodal large language models (MLLMs) for fine-grained dense prediction tasks like semantic segmentation and depth estimation without adding task-specific decoders, and it achieves competitive performance across multiple benchmarks.
Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in high-level visual understanding. However, extending these models to fine-grained dense prediction tasks, such as semantic segmentation and depth estimation, typically necessitates the incorporation of complex, task-specific decoders and other customizations. This architectural fragmentation increases model complexity and deviates from the generalist design of MLLMs, ultimately limiting their practicality. In this work, we challenge this paradigm by accommodating standard MLLMs to perform dense predictions without requiring additional task-specific decoders. The proposed model is called DenseMLLM, grounded in the standard architecture with a novel vision token supervision strategy for multiple labels and tasks. Despite its minimalist design, our model achieves highly competitive performance across a wide range of dense prediction and vision-language benchmarks, demonstrating that a standard, general-purpose MLLM can effectively support dense perception without architectural specialization.