Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework
This work tackles the problem of language bias and sensitivity in LVLMs for researchers and practitioners developing robust multi-modal AI systems, offering an incremental improvement in robustness.
This paper addresses language bias and sensitivity in Large Vision-Language Models (LVLMs) by proposing the Self-Critical Inference (SCI) framework. SCI uses multi-round counterfactual reasoning with textual and visual perturbations to improve robustness, outperforming baseline methods on a new dynamic benchmark.
The emergence of Large Language Models (LLMs) has driven rapid progress in multi-modal learning, particularly in the development of Large Vision-Language Models (LVLMs). However, existing LVLM training paradigms place excessive reliance on the LLM component, giving rise to two critical robustness challenges: language bias and language sensitivity. To address both issues simultaneously, we propose a novel Self-Critical Inference (SCI) framework that extends Visual Contrastive Decoding by conducting multi-round counterfactual reasoning through both textual and visual perturbations. This process further introduces a new strategy for improving robustness by scaling the number of counterfactual rounds. Moreover, we also observe that failure cases of LVLMs differ significantly across models, indicating that fixed robustness benchmarks may not be able to capture the true reliability of LVLMs. To this end, we propose the Dynamic Robustness Benchmark (DRBench), a model-specific evaluation framework targeting both language bias and sensitivity issues. Extensive experiments show that SCI consistently outperforms baseline methods on DRBench, and that increasing the number of inference rounds further boosts robustness beyond existing single-step counterfactual reasoning methods.