Beyond Single Frames: Can LMMs Comprehend Temporal and Contextual Narratives in Image Sequences?
This addresses a gap in multimodal AI by highlighting limitations in temporal reasoning for researchers and developers, though it is incremental as it focuses on benchmarking rather than solving the problem.
The paper tackles the problem of evaluating Large Multimodal Models (LMMs) on understanding sequential images, introducing the StripCipher benchmark and finding that LMMs like GPT-4o perform poorly, with only 23.93% accuracy in reordering tasks compared to 80% human performance.
Large Multimodal Models (LMMs) have achieved remarkable success across various visual-language tasks. However, existing benchmarks predominantly focus on single-image understanding, leaving the analysis of image sequences largely unexplored. To address this limitation, we introduce StripCipher, a comprehensive benchmark designed to evaluate capabilities of LMMs to comprehend and reason over sequential images. StripCipher comprises a human-annotated dataset and three challenging subtasks: visual narrative comprehension, contextual frame prediction, and temporal narrative reordering. Our evaluation of 16 state-of-the-art LMMs, including GPT-4o and Qwen2.5VL, reveals a significant performance gap compared to human capabilities, particularly in tasks that require reordering shuffled sequential images. For instance, GPT-4o achieves only 23.93% accuracy in the reordering subtask, which is 56.07% lower than human performance. Further quantitative analysis discuss several factors, such as input format of images, affecting the performance of LLMs in sequential understanding, underscoring the fundamental challenges that remain in the development of LMMs.