Visual CoT Makes VLMs Smarter but More Fragile
This work addresses the fragility of advanced reasoning methods in multimodal AI for researchers and practitioners, though it is incremental as it builds on existing Visual CoT paradigms.
The paper systematically evaluates the robustness of Visual Chain-of-Thought (CoT) techniques in Vision-Language Models under image perturbations, finding that while Visual CoT improves accuracy by 5-15% on clean and corrupted images across 12 corruption types and 4 VQA datasets, it also increases sensitivity to noise, causing sharper performance degradation compared to standard VLMs.
Chain-of-Thought (CoT) techniques have significantly enhanced reasoning in Vision-Language Models (VLMs). Extending this paradigm, Visual CoT integrates explicit visual edits, such as cropping or annotating regions of interest, into the reasoning process, achieving superior multimodal performance. However, the robustness of Visual CoT-based VLMs against image-level noise remains unexplored. In this paper, we present the first systematic evaluation of Visual CoT robustness under visual perturbations. Our benchmark spans 12 image corruption types across 4 Visual Question Answering (VQA) datasets, enabling a comprehensive comparison between VLMs that use Visual CoT, and VLMs that do not. The results reveal that integrating Visual CoT consistently improves absolute accuracy regardless of whether the input images are clean or corrupted by noise; however, it also increases sensitivity to input perturbations, resulting in sharper performance degradation compared to standard VLMs. Through extensive analysis, we identify the intermediate reasoning components of Visual CoT, i.e., the edited image patches , as the primary source of fragility. Building on this analysis, we propose a plug-and-play robustness enhancement method that integrates Grounding DINO model into the Visual CoT pipeline, providing high-confidence local visual cues to stabilize reasoning. Our work reveals clear fragility patterns in Visual CoT and offers an effective, architecture-agnostic solution for enhancing visual robustness.