Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models
This work addresses a critical gap in tool-calling robustness for multilingual users, though it is incremental as it builds on existing evaluations.
The paper tackled the problem of multilingual robustness in tool calling for large language models, finding that models often fail due to parameter value language mismatch, and while inference-time strategies reduce errors, they cannot fully recover English-level performance.
Large Language Models (LLMs) are increasingly deployed as agents that invoke external tools through structured function calls. While recent work reports strong tool-calling performance under standard English-centric evaluations, the robustness of tool calling under multilingual user interactions remains underexplored. In this work, we introduce MLCL, a diagnostic benchmark, and conduct a systematic evaluation of multilingual tool calling across Chinese, Hindi, and the low-resource language Igbo. Through fine-grained error analysis, we show that many failures occur despite correct intent understanding and tool selection. We identify parameter value language mismatch as a dominant failure mode, where models generate semantically appropriate parameter values in the user's language, violating language-invariant execution conventions. We further evaluate several inference-time system strategies and find that while these strategies substantially reduce language-induced execution errors, none of them can fully recover English-level performance.