Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings
This work addresses the challenge of creating more accurate and wide-coverage word embeddings for natural language processing tasks, but it is incremental as it builds on existing concatenation baselines.
The paper tackled the problem of learning meta word embeddings from multiple source embeddings by proposing unsupervised weighted concatenation methods, showing that these methods outperform previous meta-embedding approaches on benchmark datasets.
Given multiple source word embeddings learnt using diverse algorithms and lexical resources, meta word embedding learning methods attempt to learn more accurate and wide-coverage word embeddings. Prior work on meta-embedding has repeatedly discovered that simple vector concatenation of the source embeddings to be a competitive baseline. However, it remains unclear as to why and when simple vector concatenation can produce accurate meta-embeddings. We show that weighted concatenation can be seen as a spectrum matching operation between each source embedding and the meta-embedding, minimising the pairwise inner-product loss. Following this theoretical analysis, we propose two \emph{unsupervised} methods to learn the optimal concatenation weights for creating meta-embeddings from a given set of source embeddings. Experimental results on multiple benchmark datasets show that the proposed weighted concatenated meta-embedding methods outperform previously proposed meta-embedding learning methods.