CURVE: A Benchmark for Cultural and Multilingual Long Video Reasoning
This addresses the problem of cultural and linguistic bias in video AI evaluation for researchers and developers, representing a significant step beyond incremental improvements by providing a new benchmark.
The authors tackled the bias in video understanding benchmarks by introducing CURVE, a multicultural and multilingual video reasoning benchmark with human-generated annotations across 18 locales, revealing that state-of-the-art Video-LLMs perform substantially below human-level accuracy, with errors primarily in visual perception of cultural elements.
Recent advancements in video models have shown tremendous progress, particularly in long video understanding. However, current benchmarks predominantly feature western-centric data and English as the dominant language, introducing significant biases in evaluation. To address this, we introduce CURVE (Cultural Understanding and Reasoning in Video Evaluation), a challenging benchmark for multicultural and multilingual video reasoning. CURVE comprises high-quality, entirely human-generated annotations from diverse, region-specific cultural videos across 18 global locales. Unlike prior work that relies on automatic translations, CURVE provides complex questions, answers, and multi-step reasoning steps, all crafted in native languages. Making progress on CURVE requires a deeply situated understanding of visual cultural context. Furthermore, we leverage CURVE's reasoning traces to construct evidence-based graphs and propose a novel iterative strategy using these graphs to identify fine-grained errors in reasoning. Our evaluations reveal that SoTA Video-LLMs struggle significantly, performing substantially below human-level accuracy, with errors primarily stemming from the visual perception of cultural elements. CURVE will be publicly available under https://github.com/google-deepmind/neptune?tab=readme-ov-file\#minerva-cultural