Decoupling Vision and Language: Codebook Anchored Visual Adaptation
This addresses the issue of representation errors in domain-specific tasks like medical imaging for users of LVLMs, offering a decoupled adaptation approach that is incremental over existing tuning methods.
The paper tackled the problem of large vision-language models underperforming in domain-specific visual tasks by introducing CRAFT, a method that fine-tunes the vision encoder using a discrete codebook to decouple it from the language model, achieving an average gain of 13.51% across 10 benchmarks.
Large Vision-Language Models (LVLMs) use their vision encoders to translate images into representations for downstream reasoning, but the encoders often underperform in domain-specific visual tasks such as medical image diagnosis or fine-grained classification, where representation errors can cascade through the language model, leading to incorrect responses. Existing adaptation methods modify the continuous feature interface between encoder and language model through projector tuning or other parameter-efficient updates, which still couples the two components and requires re-alignment whenever the encoder changes. We introduce CRAFT (Codebook RegulAted Fine-Tuning), a lightweight method that fine-tunes the encoder using a discrete codebook that anchors visual representations to a stable token space, achieving domain adaptation without modifying other parts of the model. This decoupled design allows the adapted encoder to seamlessly boost the performance of LVLMs with different language architectures, as long as they share the same codebook. Empirically, CRAFT achieves an average gain of 13.51% across 10 domain-specific benchmarks such as VQARAD and PlantVillage, while preserving the LLM's linguistic capabilities and outperforming peer methods that operate on continuous tokens.