The Effect of Scripts and Formats on LLM Numeracy
This addresses an overlooked challenge in multilingual numerical reasoning for users of LLMs, providing actionable insights for reliable interpretation across diverse numeral scripts and formatting styles, though it is incremental in nature.
The study tackled the problem of LLM performance degradation when numerical inputs use underrepresented scripts or formats, showing that accuracy drops substantially but can be improved with targeted prompting strategies like few-shot prompting and explicit numeral mapping.
Large language models (LLMs) have achieved impressive proficiency in basic arithmetic, rivaling human-level performance on standard numerical tasks. However, little attention has been given to how these models perform when numerical expressions deviate from the prevailing conventions present in their training corpora. In this work, we investigate numerical reasoning across a wide range of numeral scripts and formats. We show that LLM accuracy drops substantially when numerical inputs are rendered in underrepresented scripts or formats, despite the underlying mathematical reasoning being identical. We further demonstrate that targeted prompting strategies, such as few-shot prompting and explicit numeral mapping, can greatly narrow this gap. Our findings highlight an overlooked challenge in multilingual numerical reasoning and provide actionable insights for working with LLMs to reliably interpret, manipulate, and generate numbers across diverse numeral scripts and formatting styles.