SigLIP-HD by Fine-to-Coarse Supervision
This work addresses the problem of achieving fine visual perception in multimodal LLMs without the computational overhead of high-resolution images, offering a practical improvement for resource-constrained settings.
SigLIP-HD improves visual representation quality in multimodal LLMs by using a fine-to-coarse supervision method that forces coarse features from mid-resolution images to mimic fine-grained features from high-resolution versions, achieving better performance on MLLM benchmarks (especially OCR tasks) without increasing inference cost.
High-quality visual representation is a long-standing pursuit in computer vision. In the context of multimodal LLMs (MLLMs), feeding higher-resolution images can produce more fine-grained visual tokens. However, it introduces additional computational and design complexity, due to multiple forward passes and post-processing of increased tokens. Before simply adopting a higher resolution, have we truly unlocked the model's full perception capability at a standard resolution? Therefore, we study an interesting problem: how to achieve fine visual perception under lower cost without larger images. We present SigLIP-HD in this work. The core is a highly simple fine-to-coarse supervision design. We enforce the coarse feature of a mid-resolution image to mimic the fine-grained feature of its high-resolution version. We build this framework on the advanced SigLIP 2 model. Our final model produces better visual tokens at exactly the same inference budget. It is validated on extensive MLLM benchmarks and consistently delivers stronger results than our baseline model, especially on OCR-related tasks.