A Generative Model of Words and Relationships from Multiple Sources
This addresses the challenge of data scarcity in specialized domains for researchers and practitioners in NLP, though it is incremental as it builds on existing embedding methods.
The paper tackles the problem of learning neural language models in specialized domains with limited training data by proposing a generative model that integrates evidence from diverse sources like knowledge bases, generalizing co-occurrence to include other relationships as affine transformations. The result shows effectiveness in outperforming recent models on link prediction and benefiting from partially or fully unobserved data.
Neural language models are a powerful tool to embed words into semantic vector spaces. However, learning such models generally relies on the availability of abundant and diverse training examples. In highly specialised domains this requirement may not be met due to difficulties in obtaining a large corpus, or the limited range of expression in average use. Such domains may encode prior knowledge about entities in a knowledge base or ontology. We propose a generative model which integrates evidence from diverse data sources, enabling the sharing of semantic information. We achieve this by generalising the concept of co-occurrence from distributional semantics to include other relationships between entities or words, which we model as affine transformations on the embedding space. We demonstrate the effectiveness of this approach by outperforming recent models on a link prediction task and demonstrating its ability to profit from partially or fully unobserved data training labels. We further demonstrate the usefulness of learning from different data sources with overlapping vocabularies.