Comparative Study of Multilingual Idioms and Similes in Large Language Models
This work addresses the gap in evaluating LLMs on multilingual figurative language interpretation, providing insights for NLP researchers, but it is incremental as it extends existing datasets and methods.
This study compared the performance of large language models in interpreting idioms and similes across multiple languages, finding that prompt engineering effectiveness varies by figurative type, language, and model, with open-source models struggling in low-resource languages for similes and idiom interpretation nearing saturation for many languages.
This study addresses the gap in the literature concerning the comparative performance of LLMs in interpreting different types of figurative language across multiple languages. By evaluating LLMs using two multilingual datasets on simile and idiom interpretation, we explore the effectiveness of various prompt engineering strategies, including chain-of-thought, few-shot, and English translation prompts. We extend the language of these datasets to Persian as well by building two new evaluation sets. Our comprehensive assessment involves both closed-source (GPT-3.5, GPT-4o mini, Gemini 1.5), and open-source models (Llama 3.1, Qwen2), highlighting significant differences in performance across languages and figurative types. Our findings reveal that while prompt engineering methods are generally effective, their success varies by figurative type, language, and model. We also observe that open-source models struggle particularly with low-resource languages in similes. Additionally, idiom interpretation is nearing saturation for many languages, necessitating more challenging evaluations.