MMR: Evaluating Reading Ability of Large Multimodal Models
This work addresses the need for better benchmarks to assess LMMs' complex reasoning and spatial understanding in text-rich images, which is incremental as it builds on existing evaluation efforts.
The authors tackled the problem of evaluating large multimodal models (LMMs) on text-rich images by proposing the Multi-Modal Reading (MMR) benchmark with 11 diverse tasks, revealing limited capabilities of state-of-the-art LMMs like GPT-4o.
Large multimodal models (LMMs) have demonstrated impressive capabilities in understanding various types of image, including text-rich images. Most existing text-rich image benchmarks are simple extraction-based question answering, and many LMMs now easily achieve high scores. This means that current benchmarks fail to accurately reflect performance of different models, and a natural idea is to build a new benchmark to evaluate their complex reasoning and spatial understanding abilities. In this work, we propose the Multi-Modal Reading (MMR) benchmark in 11 diverse tasks to evaluate LMMs for text-rich image understanding. MMR is the first text-rich image benchmark built on human annotations with the help of language models. By evaluating several state-of-the-art LMMs, including GPT-4o, it reveals the limited capabilities of existing LMMs underscoring the value of our benchmark.