PM4Bench: A Parallel Multilingual Multi-Modal Multi-task Benchmark for Large Vision Language Model
This work addresses the need for fair and comprehensive evaluation of LVLMs across languages and modalities, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the limitations of existing multilingual benchmarks for Large Vision Language Models (LVLMs) by proposing PM4Bench, a parallel multilingual multi-modal multi-task benchmark, and used it to evaluate 11 mainstream LVLMs, revealing significant cross-linguistic performance disparities, particularly in vision settings, with OCR capability identified as a key factor.
Existing multilingual benchmarks for Large Vision Language Models (LVLMs) suffer from limitations including language-specific content biases, disjointed multimodal input formats, and a lack of safety evaluation. To address these gaps, we propose PM4Bench, the first Parallel Multilingual Multi-Modal Multi-task Benchmark for LVLMs. PM4Bench features a parallel corpus design across 10 languages, enabling fair and accurate cross-lingual comparisons. It includes the vision setting where text and queries are embedded in images, requiring LVLMs to simultaneously "see", "read", and "think", aligning with real-world applications. Additionally, PM\textsuperscript{4}Bench incorporates safety evaluations, addressing critical oversight in existing multilingual benchmarks. Using PM4Bench, we evaluate 11 mainstream LVLMs, revealing significant cross-linguistic performance disparities, particularly in vision settings, and identifying OCR capability as a key determinant of these imbalances. We will release PM4Bench at https://github.com/opendatalab/PM4Bench .