A Visual Leap in CLIP Compositionality Reasoning through Generation of Counterfactual Sets
This addresses a key bottleneck in vision-language models for tasks requiring compositional reasoning, offering a novel data generation approach.
The paper tackles the problem of vision-language models struggling with compositional reasoning by proposing a block-based diffusion method to automatically generate counterfactual datasets, which significantly improves visual reasoning performance and achieves state-of-the-art results across multiple benchmarks with less training data.
Vision-language models (VLMs) often struggle with compositional reasoning due to insufficient high-quality image-text data. To tackle this challenge, we propose a novel block-based diffusion approach that automatically generates counterfactual datasets without manual annotation. Our method utilizes large language models to identify entities and their spatial relationships. It then independently generates image blocks as "puzzle pieces" coherently arranged according to specified compositional rules. This process creates diverse, high-fidelity counterfactual image-text pairs with precisely controlled variations. In addition, we introduce a specialized loss function that differentiates inter-set from intra-set samples, enhancing training efficiency and reducing the need for negative samples. Experiments demonstrate that fine-tuning VLMs with our counterfactual datasets significantly improves visual reasoning performance. Our approach achieves state-of-the-art results across multiple benchmarks while using substantially less training data than existing methods.