MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models
This addresses the issue of unreliable video descriptions for users of LVMs, but it is incremental as it focuses on benchmarking rather than solving hallucinations directly.
The paper tackles the problem of hallucinations in Large Video Models (LVMs) by introducing MESH, a benchmark that systematically evaluates these inaccuracies using a Question-Answering framework aligned with human video understanding, showing that LVMs are prone to hallucinations with fine details or multiple actions in longer videos.
Large Video Models (LVMs) build on the semantic capabilities of Large Language Models (LLMs) and vision modules by integrating temporal information to better understand dynamic video content. Despite their progress, LVMs are prone to hallucinations-producing inaccurate or irrelevant descriptions. Current benchmarks for video hallucination depend heavily on manual categorization of video content, neglecting the perception-based processes through which humans naturally interpret videos. We introduce MESH, a benchmark designed to evaluate hallucinations in LVMs systematically. MESH uses a Question-Answering framework with binary and multi-choice formats incorporating target and trap instances. It follows a bottom-up approach, evaluating basic objects, coarse-to-fine subject features, and subject-action pairs, aligning with human video understanding. We demonstrate that MESH offers an effective and comprehensive approach for identifying hallucinations in videos. Our evaluations show that while LVMs excel at recognizing basic objects and features, their susceptibility to hallucinations increases markedly when handling fine details or aligning multiple actions involving various subjects in longer videos.