Evaluating Robustness of Large Language Models Against Multilingual Typographical Errors
This work addresses the robustness of LLMs for real-world multilingual applications, but it is incremental as it builds on existing evaluation frameworks.
The paper tackled the problem of large language models' vulnerability to multilingual typographical errors by introducing MulTypo, a typo generation algorithm, and found that typos consistently degrade performance, especially in generative and reasoning tasks, with language-dependent effects such as high-resource languages being more robust.
Large language models (LLMs) are increasingly deployed in multilingual, real-world applications with user inputs -- naturally introducing typographical errors (typos). Yet most benchmarks assume clean input, leaving the robustness of LLMs to typos across languages largely underexplored. To address this gap, we introduce MulTypo, a multilingual typo generation algorithm that simulates human-like errors based on language-specific keyboard layouts and typing behavior. We evaluate 18 open-source LLMs across three model families and five downstream tasks spanning language inference, multi-choice question answering, mathematical reasoning, and machine translation tasks. Our results show that typos consistently degrade performance, particularly in generative tasks and those requiring reasoning -- while the natural language inference task is comparatively more robust. Instruction tuning improves clean-input performance but may increase brittleness under noise. We also observe language-dependent robustness: high-resource languages are generally more robust than low-resource ones, and translation from English is more robust than translation into English. Our findings underscore the need for noise-aware training and multilingual robustness evaluation. We make our code and data publicly available.