Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping
This addresses fine-grained perceptual grounding issues in multimodal large language models, offering a test-time enhancement that is incremental but effective for improving performance on tasks like VQA and reasoning.
The paper tackles the problem of MLLMs missing small details and spatial relations in cluttered scenes by introducing AttWarp, a lightweight method that uses cross-modal attention to warp input images, reallocating resolution to query-relevant areas without changing model weights, resulting in improved accuracy across five benchmarks and four MLLMs.
Multimodal large language models (MLLMs) often miss small details and spatial relations in cluttered scenes, leading to errors in fine-grained perceptual grounding. We introduce AttWarp, a lightweight method that allocates more resolution to query-relevant content while compressing less informative areas, all while preserving global context. At test time, the approach uses an MLLM's cross-modal attention to perform rectilinear warping of the input image, reallocating spatial resolution toward regions the model deems important, without changing model weights or architecture. This attention-guided warping preserves all original image information but redistributes it non-uniformly, so small objects and subtle relationships become easier for the same model to read while the global layout remains intact. Across five benchmarks (TextVQA, GQA, DocVQA, POPE, MMMU) and four MLLMs (LLaVA, Qwen-VL, InternVL, and InstructBLIP), AttWarp consistently improves accuracy, strengthens compositional reasoning, and reduces hallucinations, outperforming four competitive baselines that manipulate raw images at test time. Together, these results show that attention-guided warping prioritizes information relevant to the query while preserving context, and that the same MLLMs perform better when given such warped inputs.