Investigating Bias: A Multilingual Pipeline for Generating, Solving, and Evaluating Math Problems with LLMs
This work addresses the issue of equitable AI in education for multilingual users, but it is incremental as it applies existing methods to new data and languages.
The paper tackled the problem of linguistic bias in LLMs for educational math support by developing an automated multilingual pipeline to generate, solve, and evaluate 628 math problems in English, German, and Arabic, finding that English solutions consistently ranked highest while Arabic often scored lower, highlighting a performance gap.
Large Language Models (LLMs) are increasingly used for educational support, yet their response quality varies depending on the language of interaction. This paper presents an automated multilingual pipeline for generating, solving, and evaluating math problems aligned with the German K-10 curriculum. We generated 628 math exercises and translated them into English, German, and Arabic. Three commercial LLMs (GPT-4o-mini, Gemini 2.5 Flash, and Qwen-plus) were prompted to produce step-by-step solutions in each language. A held-out panel of LLM judges, including Claude 3.5 Haiku, evaluated solution quality using a comparative framework. Results show a consistent gap, with English solutions consistently rated highest, and Arabic often ranked lower. These findings highlight persistent linguistic bias and the need for more equitable multilingual AI systems in education.