L2V-CoT: Cross-Modal Transfer of Chain-of-Thought Reasoning via Latent Intervention
This addresses a bottleneck in vision-language models for multi-step reasoning tasks, offering a training-free solution that is incremental but effective.
The paper tackles the problem of transferring Chain-of-Thought reasoning from large language models to vision-language models to improve their multi-step reasoning, achieving performance that surpasses both training-free baselines and supervised methods.
Recently, Chain-of-Thought (CoT) reasoning has significantly enhanced the capabilities of large language models (LLMs), but Vision-Language Models (VLMs) still struggle with multi-step reasoning tasks due to limited multimodal reasoning data. To bridge this gap, researchers have explored methods to transfer CoT reasoning from LLMs to VLMs. However, existing approaches either need high training costs or require architectural alignment. In this paper, we use Linear Artificial Tomography (LAT) to empirically show that LLMs and VLMs share similar low-frequency latent representations of CoT reasoning despite architectural differences. Based on this insight, we propose L2V-CoT, a novel training-free latent intervention approach that transfers CoT reasoning from LLMs to VLMs. L2V-CoT extracts and resamples low-frequency CoT representations from LLMs in the frequency domain, enabling dimension matching and latent injection into VLMs during inference to enhance reasoning capabilities. Extensive experiments demonstrate that our approach consistently outperforms training-free baselines and even surpasses supervised methods.