A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models
This work addresses the challenge of figurative language understanding for natural language processing applications, but it is incremental as it focuses on a specific task and dataset.
The authors tackled the problem of idiomatic language processing by introducing IdioTS, a dataset designed by experts to evaluate LLMs on idiom detection, and conducted a comprehensive evaluation with error analysis.
In this work, we explore idiomatic language processing with Large Language Models (LLMs). We introduce the Idiomatic language Test Suite IdioTS, a new dataset of difficult examples specifically designed by language experts to assess the capabilities of LLMs to process figurative language at sentence level. We propose a comprehensive evaluation methodology based on an idiom detection task, where LLMs are prompted with detecting an idiomatic expression in a given English sentence. We present a thorough automatic and manual evaluation of the results and an extensive error analysis.