Retrofitting Contextualized Word Embeddings with Paraphrases
This addresses robustness issues in NLP models for tasks like classification and inference, but it is incremental as it builds on existing embedding methods.
The paper tackled the problem of contextualized word embeddings being unstable under paraphrasing, which reduces robustness in downstream tasks, and proposed a retrofitting method that significantly improved performance over the original ELMo on sentence classification and language inference tasks.
Contextualized word embedding models, such as ELMo, generate meaningful representations of words and their context. These models have been shown to have a great impact on downstream applications. However, in many cases, the contextualized embedding of a word changes drastically when the context is paraphrased. As a result, the downstream model is not robust to paraphrasing and other linguistic variations. To enhance the stability of contextualized word embedding models, we propose an approach to retrofitting contextualized embedding models with paraphrase contexts. Our method learns an orthogonal transformation on the input space, which seeks to minimize the variance of word representations on paraphrased contexts. Experiments show that the retrofitted model significantly outperforms the original ELMo on various sentence classification and language inference tasks.