SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs
This addresses the need for scalable and fine-grained hallucination evaluation in LVLMs, though it is incremental as it builds on prior work by automating data construction and expanding categorization.
The authors tackled the problem of evaluating hallucinations in Large Vision-Language Models by proposing SHALE, a scalable benchmark that assesses both faithfulness and factuality hallucinations with fine-grained categorization, revealing significant factuality issues and high sensitivity to perturbations in over 20 models.
Despite rapid advances, Large Vision-Language Models (LVLMs) still suffer from hallucinations, i.e., generating content inconsistent with input or established world knowledge, which correspond to faithfulness and factuality hallucinations, respectively. Prior studies primarily evaluate faithfulness hallucination at a rather coarse level (e.g., object-level) and lack fine-grained analysis. Additionally, existing benchmarks often rely on costly manual curation or reused public datasets, raising concerns about scalability and data leakage. To address these limitations, we propose an automated data construction pipeline that produces scalable, controllable, and diverse evaluation data. We also design a hierarchical hallucination induction framework with input perturbations to simulate realistic noisy scenarios. Integrating these designs, we construct SHALE, a Scalable HALlucination Evaluation benchmark designed to assess both faithfulness and factuality hallucinations via a fine-grained hallucination categorization scheme. SHALE comprises over 30K image-instruction pairs spanning 12 representative visual perception aspects for faithfulness and 6 knowledge domains for factuality, considering both clean and noisy scenarios. Extensive experiments on over 20 mainstream LVLMs reveal significant factuality hallucinations and high sensitivity to semantic perturbations.