Visual Instruction Pretraining for Domain-Specific Foundation Models
This work addresses the problem of incomplete perception-reasoning loops in computer vision for domain-specific applications like remote sensing and medical imaging, representing a novel paradigm rather than an incremental improvement.
The paper tackles the gap in integrating high-level reasoning with low-level perceptual feature learning in computer vision by proposing Visual Instruction Pretraining (ViTP), a new paradigm that embeds a Vision Transformer in a Vision-Language Model and uses visual instruction data for pretraining, achieving state-of-the-art performance on 16 remote sensing and medical imaging benchmarks.
Modern computer vision is converging on a closed loop in which perception, reasoning and generation mutually reinforce each other. However, this loop remains incomplete: the top-down influence of high-level reasoning on the foundational learning of low-level perceptual features is not yet underexplored. This paper addresses this gap by proposing a new paradigm for pretraining foundation models in downstream domains. We introduce Visual insTruction Pretraining (ViTP), a novel approach that directly leverages reasoning to enhance perception. ViTP embeds a Vision Transformer (ViT) backbone within a Vision-Language Model and pretrains it end-to-end using a rich corpus of visual instruction data curated from target downstream domains. ViTP is powered by our proposed Visual Robustness Learning (VRL), which compels the ViT to learn robust and domain-relevant features from a sparse set of visual tokens. Extensive experiments on 16 challenging remote sensing and medical imaging benchmarks demonstrate that ViTP establishes new state-of-the-art performance across a diverse range of downstream tasks. The code is available at https://github.com/zcablii/ViTP.