Neural Embeddings for Text
This addresses the challenge of creating more meaningful text representations for natural language processing tasks, but it appears incremental as it builds on existing language model techniques.
The authors tackled the problem of representing semantic meaning in text by proposing neural embeddings derived from the weights of a language model's neurons, and they confirmed its ability to reflect semantics through analysis on datasets and comparison with state-of-the-art sentence embeddings.
We propose a new kind of embedding for natural language text that deeply represents semantic meaning. Standard text embeddings use the outputs from hidden layers of a pretrained language model. In our method, we let a language model learn from the text and then literally pick its brain, taking the actual weights of the model's neurons to generate a vector. We call this representation of the text a neural embedding. We confirm the ability of this representation to reflect semantics of the text by an analysis of its behavior on several datasets, and by a comparison of neural embedding with state of the art sentence embeddings.