Updating Pre-trained Word Vectors and Text Classifiers using Monolingual Alignment
This addresses the challenge of adapting general-purpose NLP models to changing word distributions, though it appears incremental as it builds on existing alignment techniques.
The paper tackles the problem of adapting pre-trained word vector models to new textual data with different language distributions by framing it as a monolingual alignment problem and averaging models after alignment using the RCSLS criterion. The approach yields good performance in word embedding and text classification applications, outperforming fine-tuning baselines.
In this paper, we focus on the problem of adapting word vector-based models to new textual data. Given a model pre-trained on large reference data, how can we adapt it to a smaller piece of data with a slightly different language distribution? We frame the adaptation problem as a monolingual word vector alignment problem, and simply average models after alignment. We align vectors using the RCSLS criterion. Our formulation results in a simple and efficient algorithm that allows adapting general-purpose models to changing word distributions. In our evaluation, we consider applications to word embedding and text classification models. We show that the proposed approach yields good performance in all setups and outperforms a baseline consisting in fine-tuning the model on new data.