Frustratingly Easy Meta-Embedding -- Computing Meta-Embeddings by Averaging Source Word Embeddings
This provides a simple and effective solution for NLP practitioners needing to combine embeddings, though it is incremental as it builds on existing meta-embedding research.
The paper tackles the problem of creating meta-embeddings from pre-trained word embeddings by showing that simply averaging two distinct sets yields performance comparable or better than more complex methods, despite the incomparability of the source vector spaces.
Creating accurate meta-embeddings from pre-trained source embeddings has received attention lately. Methods based on global and locally-linear transformation and concatenation have shown to produce accurate meta-embeddings. In this paper, we show that the arithmetic mean of two distinct word embedding sets yields a performant meta-embedding that is comparable or better than more complex meta-embedding learning methods. The result seems counter-intuitive given that vector spaces in different source embeddings are not comparable and cannot be simply averaged. We give insight into why averaging can still produce accurate meta-embedding despite the incomparability of the source vector spaces.