Investigating the interaction of linguistic and mathematical reasoning in language models using multilingual number puzzles
This highlights a key limitation in LLMs' reasoning abilities for tasks requiring implicit linguistic-mathematical inference, which is incremental as it builds on known challenges in model reasoning.
The study investigated why large language models (LLMs) struggle with multilingual number puzzles involving cross-linguistic numeral systems, finding that models fail unless mathematical operations are explicitly marked with symbols like '+', unlike humans who infer implicit compositional structures.
Across languages, numeral systems vary widely in how they construct and combine numbers. While humans consistently learn to navigate this diversity, large language models (LLMs) struggle with linguistic-mathematical puzzles involving cross-linguistic numeral systems, which humans can learn to solve successfully. We investigate why this task is difficult for LLMs through a series of experiments that untangle the linguistic and mathematical aspects of numbers in language. Our experiments establish that models cannot consistently solve such problems unless the mathematical operations in the problems are explicitly marked using known symbols ($+$, $\times$, etc., as in "twenty + three"). In further ablation studies, we probe how individual parameters of numeral construction and combination affect performance. While humans use their linguistic understanding of numbers to make inferences about the implicit compositional structure of numerals, LLMs seem to lack this notion of implicit numeral structure. We conclude that the ability to flexibly infer compositional rules from implicit patterns in human-scale data remains an open challenge for current reasoning models.