IBERT: Idiom Cloze-style reading comprehension with Attention
This addresses the challenge of understanding non-compositional idioms in NLP, which is important for improving reading comprehension in casual and literary texts, though it appears incremental as it builds on existing BERT and Seq2Seq approaches.
The paper tackled the idiom cloze task in NLP by proposing a BERT-based Seq2Seq model that encodes idiomatic expressions with attention to both local and global context, achieving better performance than existing state-of-the-art methods on the EPIE Static Corpus dataset.
Idioms are special fixed phrases usually derived from stories. They are commonly used in casual conversations and literary writings. Their meanings are usually highly non-compositional. The idiom cloze task is a challenge problem in Natural Language Processing (NLP) research problem. Previous approaches to this task are built on sequence-to-sequence (Seq2Seq) models and achieved reasonably well performance on existing datasets. However, they fall short in understanding the highly non-compositional meaning of idiomatic expressions. They also do not consider both the local and global context at the same time. In this paper, we proposed a BERT-based embedding Seq2Seq model that encodes idiomatic expressions and considers them in both global and local context. Our model uses XLNET as the encoder and RoBERTa for choosing the most probable idiom for a given context. Experiments on the EPIE Static Corpus dataset show that our model performs better than existing state-of-the-arts.