Discovering linguistic (ir)regularities in word embeddings through max-margin separating hyperplanes
This work addresses the challenge of accurately modeling word relationships in NLP, but it is incremental as it builds on prior offset-based methods with a new linear separability assumption.
The authors tackled the problem of learning relationships between words in embedding spaces by showing that offsets between related tokens are closer to orthogonal than parallel, and they proposed using max-margin hyperplanes to separate target tokens, achieving the best results in discovering linguistic regularities across various embedding models and context choices.
We experiment with new methods for learning how related words are positioned relative to each other in word embedding spaces. Previous approaches learned constant vector offsets: vectors that point from source tokens to target tokens with an assumption that these offsets were parallel to each other. We show that the offsets between related tokens are closer to orthogonal than parallel, and that they have low cosine similarities. We proceed by making a different assumption; target tokens are linearly separable from source and un-labeled tokens. We show that a max-margin hyperplane can separate target tokens and that vectors orthogonal to this hyperplane represent the relationship between source and targets. We find that this representation of the relationship obtains the best results in dis-covering linguistic regularities. We experiment with vector space models trained by a variety of algorithms (Word2vec: CBOW/skip-gram, fastText, or GloVe), and various word context choices such as linear word-order, syntax dependency grammars, and with and without knowledge of word position. These experiments show that our model, SVMCos, is robust to a range of experimental choices when training word embeddings.