Mind the (Language) Gap: Towards Probing Numerical and Cross-Lingual Limits of LVLMs
This work addresses limitations in LVLMs for structure-aware visual text understanding, numerical reasoning, and cross-lingual generalization, which is incremental as it provides a new benchmark for a specific domain.
The authors tackled the problem of evaluating large vision-language models (LVLMs) on complex numerical and cross-lingual reasoning by introducing MMCRICBENCH-3K, a benchmark for visual question answering on cricket scorecards, and found that state-of-the-art models like GPT-4o and Qwen2.5VL struggle on both English and Hindi subsets, with performance dropping further on Hindi.
We introduce MMCRICBENCH-3K, a benchmark for Visual Question Answering (VQA) on cricket scorecards, designed to evaluate large vision-language models (LVLMs) on complex numerical and cross-lingual reasoning over semi-structured tabular images. MMCRICBENCH-3K comprises 1,463 synthetically generated scorecard images from ODI, T20, and Test formats, accompanied by 1,500 English QA pairs. It includes two subsets: MMCRICBENCH-E-1.5K, featuring English scorecards, and MMCRICBENCH-H-1.5K, containing visually similar Hindi scorecards, with all questions and answers kept in English to enable controlled cross-script evaluation. The task demands reasoning over structured numerical data, multi-image context, and implicit domain knowledge. Empirical results show that even state-of-the-art LVLMs, such as GPT-4o and Qwen2.5VL, struggle on the English subset despite it being their primary training language and exhibit a further drop in performance on the Hindi subset. This reveals key limitations in structure-aware visual text understanding, numerical reasoning, and cross-lingual generalization. The dataset is publicly available via Hugging Face at https://huggingface.co/datasets/DIALab/MMCricBench, to promote LVLM research in this direction.