MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation
This addresses the need for more robust and realistic benchmarks in multilingual mathematical reasoning evaluation, particularly for low-resource languages, though it is incremental as it extends existing datasets and methods.
The paper tackled the problem of evaluating multilingual mathematical reasoning in large language models by introducing MGSM-Pro, an extension of the MGSM dataset with GSM-Symbolic approach, and found that many low-resource languages suffer large performance drops (e.g., up to 20% in some cases) when tested on digit instantiations different from the original set, with proprietary models like Gemini 2.5 Flash and GPT-4.1 showing less robustness.
Large language models have made substantial progress in mathematical reasoning. However, benchmark development for multilingual evaluation has lagged behind English in both difficulty and recency. Recently, GSM-Symbolic showed a strong evidence of high variance when models are evaluated on different instantiations of the same question; however, the evaluation was conducted only in English. In this paper, we introduce MGSM-Pro, an extension of MGSM dataset with GSM-Symbolic approach. Our dataset provides five instantiations per MGSM question by varying names, digits and irrelevant context. Evaluations across nine languages reveal that many low-resource languages suffer large performance drops when tested on digit instantiations different from those in the original test set. We further find that some proprietary models, notably Gemini 2.5 Flash and GPT-4.1, are less robust to digit instantiation, whereas Claude 4.0 Sonnet is more robust. Among open models, GPT-OSS 120B and DeepSeek V3 show stronger robustness. Based on these findings, we recommend evaluating each problem using at least five digit-varying instantiations to obtain a more robust and realistic assessment of math reasoning.