Language-Pretraining-Induced Bias: A Strong Foundation for General Vision Tasks
This work addresses the problem of cross-modality adaptation for researchers and practitioners in AI, offering a practical method to leverage language models in vision, though it is incremental in building on existing transfer learning concepts.
The paper tackles the challenge of adapting language-pretrained models to vision tasks by introducing a bridge training stage, showing that partial adaptation can effectively align LLM parameters with visual tasks without manual labeling.
The ratio of outlier parameters in language pre-training models and vision pre-training models differs significantly, making cross-modality (language and vision) inherently more challenging than cross-domain adaptation. As a result, many prior studies have focused on cross-domain transfer rather than attempting to bridge language and vision modalities, assuming that language pre-trained models are unsuitable for downstream visual tasks due to disparate parameter spaces. Contrary to this assumption, we show that adding a bridge training stage as a modality adaptation learner can effectively align Large Language Model (LLM) parameters with vision tasks. Specifically, we propose a simple yet powerful solution random label bridge training that requires no manual labeling and helps LLM parameters adapt to vision foundation tasks. Moreover, our findings reveal that partial bridge training is often advantageous, as certain layers in LLMs exhibit strong foundational properties that remain beneficial even without fine-tuning for visual tasks. This surprising discovery opens up new avenues for leveraging language pre-trained parameters directly within vision models and highlights the potential of partial bridge training as a practical pathway to cross-modality adaptation.