Dynamic Meta-Embeddings for Improved Sentence Representations
This addresses the challenge of embedding selection in NLP, offering a supervised learning approach that improves sentence representations, though it appears incremental as it builds on existing embedding methods.
The paper tackles the problem of selecting pre-trained word embeddings for NLP systems by introducing dynamic meta-embeddings, a method that allows neural networks to learn embedding ensembles automatically, achieving state-of-the-art performance on various tasks.
While one of the first steps in many NLP systems is selecting what pre-trained word embeddings to use, we argue that such a step is better left for neural networks to figure out by themselves. To that end, we introduce dynamic meta-embeddings, a simple yet effective method for the supervised learning of embedding ensembles, which leads to state-of-the-art performance within the same model class on a variety of tasks. We subsequently show how the technique can be used to shed new light on the usage of word embeddings in NLP systems.