Meta4XNLI: A Crosslingual Parallel Corpus for Metaphor Detection and Interpretation
This provides a new resource for researchers studying metaphor processing in language models, though it's incremental in creating a dataset rather than advancing model capabilities.
The authors tackled the problem of evaluating language models' ability to detect and interpret metaphors by creating Meta4XNLI, the first parallel dataset for Natural Language Inference annotated for metaphor detection and interpretation in English and Spanish. Their results show that fine-tuned encoders outperform decoder-only LLMs in metaphor detection, while metaphor interpretation performance notably decreases when inference is affected by metaphorical language.
Metaphors are a ubiquitous but often overlooked part of everyday language. As a complex cognitive-linguistic phenomenon, they provide a valuable means to evaluate whether language models can capture deeper aspects of meaning, including semantic, pragmatic, and cultural context. In this work, we present Meta4XNLI, the first parallel dataset for Natural Language Inference (NLI) newly annotated for metaphor detection and interpretation in both English and Spanish. Meta4XNLI facilitates the comparison of encoder- and decoder-based models in detecting and understanding metaphorical language in multilingual and cross-lingual settings. Our results show that fine-tuned encoders outperform decoders-only LLMs in metaphor detection. Metaphor interpretation is evaluated via the NLI framework with comparable performance of masked and autoregressive models, which notably decreases when the inference is affected by metaphorical language. Our study also finds that translation plays an important role in the preservation or loss of metaphors across languages, introducing shifts that might impact metaphor occurrence and model performance. These findings underscore the importance of resources like Meta4XNLI for advancing the analysis of the capabilities of language models and improving our understanding of metaphor processing across languages. Furthermore, the dataset offers previously unavailable opportunities to investigate metaphor interpretation, cross-lingual metaphor transferability, and the impact of translation on the development of multilingual annotated resources.