Not All Neural Embeddings are Born Equal
This addresses the problem of improving word embeddings for natural language processing tasks, offering a novel approach but is incremental in scope.
The paper investigated embeddings from neural machine translation models and found they outperform state-of-the-art monolingual embeddings on tasks requiring conceptual similarity and syntactic role knowledge, suggesting translation models better capture true ontological status.
Neural language models learn word representations that capture rich linguistic and conceptual information. Here we investigate the embeddings learned by neural machine translation models. We show that translation-based embeddings outperform those learned by cutting-edge monolingual models at single-language tasks requiring knowledge of conceptual similarity and/or syntactic role. The findings suggest that, while monolingual models learn information about how concepts are related, neural-translation models better capture their true ontological status.