ORIC: Benchmarking Object Recognition under Contextual Incongruity in Large Vision-Language Models
This addresses reliability issues in LVLMs for applications like robotics and visual question answering, but it is incremental as it builds on existing methods for benchmarking and fine-tuning.
The paper tackles the problem of object recognition failures in Large Vision-Language Models (LVLMs) due to contextual incongruity, where objects appear unexpectedly or are missing in atypical scenes, and introduces the ORIC framework to benchmark and improve performance, showing that fine-tuning on ORIC-style data enhances results on multiple benchmarks.
Large Vision-Language Models (LVLMs) excel at captioning, visual question answering, and robotics by combining vision and language, yet they often miss obvious objects or hallucinate nonexistent ones in atypical scenes. We examine these failures through the lens of uncertainty, focusing on contextual incongruity, where objects appear unexpectedly or fail to appear in expected contexts, and show that such cases increase recognition difficulty for state-of-the-art LVLMs. To study this regime, we introduce the Object Recognition in Incongruous Context (ORIC) framework, which constructs incongruous object-context pairs through two complementary strategies: (1) LLM-guided sampling to identify hard-to-recognize objects present in the image and (2) CLIP-guided sampling to mine plausible but absent ones. Applied to MSCOCO, ORIC produces ORIC-Bench and ORIC-style training data. Evaluating 18 LVLMs and 2 open-vocabulary detectors reveals substantial performance drops and bias patterns under incongruous contexts. Fine-tuning Qwen3-VL-8B-Instruct with Visual Reinforcement Fine-Tuning on 600 ORIC-style samples improves results on ORIC-Bench, AMBER, and HallusionBench. Overall, we show that contextual incongruity is a key source of uncertainty and provide tools for more reliable LVLMs. The code is available at https://github.com/ZhaoyangLi-1/ORIC.