3LM: Bridging Arabic, STEM, and Code through Benchmarking
This addresses a gap in Arabic LLM evaluation for underrepresented domains like STEM and code, but is incremental as it adapts existing benchmarks.
The authors tackled the lack of Arabic benchmarks for STEM and code by creating 3LM, a suite of three benchmarks including STEM questions from textbooks and synthetic sources, and a translated code benchmark, which they released publicly to support research.
Arabic is one of the most widely spoken languages in the world, yet efforts to develop and evaluate Large Language Models (LLMs) for Arabic remain relatively limited. Most existing Arabic benchmarks focus on linguistic, cultural, or religious content, leaving a significant gap in domains like STEM and code which are increasingly relevant for real-world LLM applications. To help bridge this gap, we present 3LM, a suite of three benchmarks designed specifically for Arabic. The first is a set of STEM-related question-answer pairs, naturally sourced from Arabic textbooks and educational worksheets. The second consists of synthetically generated STEM questions, created using the same sources. The third benchmark focuses on code generation, built through a careful translation of two widely used code benchmarks, incorporating a human-in-the-loop process with several rounds of review to ensure high-quality and faithful translations. We release all three benchmarks publicly to support the growth of Arabic LLM research in these essential but underrepresented areas.