CCTVBench: Contrastive Consistency Traffic VideoQA Benchmark for Multimodal LLMs
This addresses safety-critical traffic reasoning for autonomous systems, but it is incremental as it builds on existing benchmarks and methods.
The authors tackled the problem of unreliable hazard detection in traffic video reasoning by introducing CCTVBench, a benchmark that tests multimodal LLMs on paired real and counterfactual accident videos with contrastive questions, revealing a significant gap in consistency metrics and proposing C-TCD to improve performance.
Safety-critical traffic reasoning requires contrastive consistency: models must detect true hazards when an accident occurs, and reliably reject plausible-but-false hypotheses under near-identical counterfactual scenes. We present CCTVBench, a Contrastive Consistency Traffic VideoQA Benchmark built on paired real accident videos and world-model-generated counterfactual counterparts, together with minimally different, mutually exclusive hypothesis questions. CCTVBench enforces a single structured decision pattern over each video question quadruple and provides actionable diagnostics that decompose failures into positive omission, positive swap, negative hallucination, and mutual-exclusivity violation, while separating video versus question consistency. Experiments across open-source and proprietary video LLMs reveal a large and persistent gap between standard per-instance QA metrics and quadruple-level contrastive consistency, with unreliable none-of-the-above rejection as a key bottleneck. Finally, we introduce C-TCD, a contrastive decoding approach leveraging a semantically exclusive counterpart video as the contrast input at inference time, improving both instance-level QA and contrastive consistency.