All-in-one: Understanding and Generation in Multimodal Reasoning with the MAIA Benchmark
This work addresses the need for culturally specific benchmarks in multimodal AI, though it is incremental as it adapts existing evaluation frameworks to a new language and dataset.
The authors introduced MAIA, a native-Italian video benchmark to assess visual language models' reasoning abilities, finding low aggregated scores that reveal model fragility in tasks like verification and question-answering.
We introduce MAIA (Multimodal AI Assessment), a native-Italian benchmark designed for fine-grained investigation of the reasoning abilities of visual language models on videos. MAIA differs from other available video benchmarks for its design, its reasoning categories, the metric it uses, and the language and culture of the videos. MAIA evaluates Vision Language Models (VLMs) on two aligned tasks: a visual statement verification task, and an open-ended visual question-answering task, both on the same set of video-related questions. It considers twelve reasoning categories that aim to disentangle language and vision relations by highlighting the role of the visual input. Thanks to its carefully taught design, it evaluates VLMs' consistency and visually grounded natural language comprehension and generation simultaneously through an aggregated metric revealing low results that highlight models' fragility. Last but not least, the video collection has been carefully selected to reflect the Italian culture, and the language data are produced by native-speakers.