Benchmarking Sequential Visual Input Reasoning and Prediction in Multimodal Large Language Models
This provides a standardized evaluation framework for MLLMs, facilitating development of more advanced models for predictive reasoning, but it is incremental as it focuses on benchmarking rather than new methods.
The authors tackled the under-explored problem of predictive reasoning in multimodal large language models by introducing a novel benchmark that assesses capabilities in abstract pattern reasoning, human activity prediction, and physical interaction prediction, with empirical experiments confirming its soundness and revealing pros and cons of current models.
Multimodal large language models (MLLMs) have shown great potential in perception and interpretation tasks, but their capabilities in predictive reasoning remain under-explored. To address this gap, we introduce a novel benchmark that assesses the predictive reasoning capabilities of MLLMs across diverse scenarios. Our benchmark targets three important domains: abstract pattern reasoning, human activity prediction, and physical interaction prediction. We further develop three evaluation methods powered by large language model to robustly quantify a model's performance in predicting and reasoning the future based on multi-visual context. Empirical experiments confirm the soundness of the proposed benchmark and evaluation methods via rigorous testing and reveal pros and cons of current popular MLLMs in the task of predictive reasoning. Lastly, our proposed benchmark provides a standardized evaluation framework for MLLMs and can facilitate the development of more advanced models that can reason and predict over complex long sequence of multimodal input.