Distilling Monolingual and Crosslingual Word-in-Context Representations
This work addresses the need for efficient, annotation-free word representations for lexical semantics and semantic similarity, particularly in multilingual settings, though it is incremental as it builds on existing pre-trained models.
The authors tackled the problem of distilling word-in-context representations from pre-trained language models without human annotations or model updates, achieving competitive performance on monolingual lexical semantic tasks and outperforming previous methods in STS estimation, while also improving crosslingual representations.
In this study, we propose a method that distils representations of word meaning in context from a pre-trained masked language model in both monolingual and crosslingual settings. Word representations are the basis for context-aware lexical semantics and unsupervised semantic textual similarity (STS) estimation. Different from existing approaches, our method does not require human-annotated corpora nor updates of the parameters of the pre-trained model. The latter feature is appealing for practical scenarios where the off-the-shelf pre-trained model is a common asset among different applications. Specifically, our method learns to combine the outputs of different hidden layers of the pre-trained model using self-attention. Our auto-encoder based training only requires an automatically generated corpus. To evaluate the performance of the proposed approach, we performed extensive experiments using various benchmark tasks. The results on the monolingual tasks confirmed that our representations exhibited a competitive performance compared to that of the previous study for the context-aware lexical semantic tasks and outperformed it for STS estimation. The results of the crosslingual tasks revealed that the proposed method largely improved crosslingual word representations of multilingual pre-trained models.