ODI-Bench: Can MLLMs Understand Immersive Omnidirectional Environments?
This addresses the need for better evaluation of MLLMs in immersive environments like VR and AR, but it is incremental as it builds on existing MLLM and benchmark methodologies.
The paper tackles the problem of evaluating multi-modal large language models (MLLMs) on omnidirectional image understanding by introducing ODI-Bench, a benchmark with 2,000 images and 4,000 QA pairs, and finds that current models struggle, with Omni-CoT improving performance through chain-of-thought reasoning.
Omnidirectional images (ODIs) provide full 360x180 view which are widely adopted in VR, AR and embodied intelligence applications. While multi-modal large language models (MLLMs) have demonstrated remarkable performance on conventional 2D image and video understanding benchmarks, their ability to comprehend the immersive environments captured by ODIs remains largely unexplored. To address this gap, we first present ODI-Bench, a novel comprehensive benchmark specifically designed for omnidirectional image understanding. ODI-Bench contains 2,000 high-quality omnidirectional images and over 4,000 manually annotated question-answering (QA) pairs across 10 fine-grained tasks, covering both general-level and spatial-level ODI understanding. Extensive experiments are conducted to benchmark 20 representative MLLMs, including proprietary and open-source models, under both close-ended and open-ended settings. Experimental results reveal that current MLLMs still struggle to capture the immersive context provided by ODIs. To this end, we further introduce Omni-CoT, a training-free method which significantly enhances MLLMs' comprehension ability in the omnidirectional environment through chain-of-thought reasoning across both textual information and visual cues. Both the benchmark and the code will be released upon the publication.