Multimodal Word Distributions
This work addresses the need for more expressive word representations in natural language processing, though it is incremental as it builds on existing embedding methods.
The paper tackled the problem of representing words with richer semantic information by introducing multimodal word distributions using Gaussian mixtures, which outperformed existing methods like word2vec skip-grams and Gaussian embeddings on benchmark datasets such as word similarity and entailment.
Word embeddings provide point representations of words containing useful semantic information. We introduce multimodal word distributions formed from Gaussian mixtures, for multiple word meanings, entailment, and rich uncertainty information. To learn these distributions, we propose an energy-based max-margin objective. We show that the resulting approach captures uniquely expressive semantic information, and outperforms alternatives, such as word2vec skip-grams, and Gaussian embeddings, on benchmark datasets such as word similarity and entailment.