As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Language
For NLP researchers and practitioners, this work highlights a critical limitation in LLMs' understanding of nuanced language, though it is incremental as it extends existing datasets and testing methods.
The study investigates how large language models handle negation in figurative language, finding that the combination poses significant challenges and that performance heavily depends on prompt style.
Figurative language and negation are two areas that challenge current language models, however, both are widely used throughout written and spoken language. Large language models (LLMs) are also widely used in everyday contexts where they cannot necessarily be tuned for a specific dataset. It is therefore essential to understand the ability of LLMs to correctly interpret text that includes both negation and figurative language. To investigate this, we develop a set of new annotations to an existing dataset of figurative language, and test a range of language models on the dataset. We find that the combination of negation and figurativeness can present a particular challenge, and that performance overall and across different negation types is particularly dependent on the prompt style used.