Span Modeling for Idiomaticity and Figurative Language Detection with Span Contrastive Loss
This work addresses a specific issue in natural language processing for improving language model understanding of figurative expressions, representing an incremental advance with strong performance gains.
The paper tackled the problem of detecting idiomatic and figurative language, which is challenging for language models due to non-compositional phrases, by proposing BERT- and RoBERTa-based models finetuned with span contrastive loss and hard negative reweighting, achieving state-of-the-art sequence accuracy on existing datasets.
The category of figurative language contains many varieties, some of which are non-compositional in nature. This type of phrase or multi-word expression (MWE) includes idioms, which represent a single meaning that does not consist of the sum of its words. For language models, this presents a unique problem due to tokenization and adjacent contextual embeddings. Many large language models have overcome this issue with large phrase vocabulary, though immediate recognition frequently fails without one- or few-shot prompting or instruction finetuning. The best results have been achieved with BERT-based or LSTM finetuning approaches. The model in this paper contains one such variety. We propose BERT- and RoBERTa-based models finetuned with a combination of slot loss and span contrastive loss (SCL) with hard negative reweighting to improve idiomaticity detection, attaining state of the art sequence accuracy performance on existing datasets. Comparative ablation studies show the effectiveness of SCL and its generalizability. The geometric mean of F1 and sequence accuracy (SA) is also proposed to assess a model's span awareness and general performance together.