Evaluation of sentence embeddings in downstream and linguistic probing tasks
This work addresses the need for better evaluation benchmarks in sentence embeddings for NLP researchers, though it is incremental as it builds on existing methods without introducing new ones.
The paper conducted a comprehensive evaluation of recent sentence embedding methods on downstream and linguistic probing tasks, finding that a simple bag-of-words approach with deep context-dependent word embeddings outperformed sentence encoders trained on entailment datasets in many tasks, but concluded that a universal encoder performing consistently across tasks is still far from achieved.
Despite the fast developmental pace of new sentence embedding methods, it is still challenging to find comprehensive evaluations of these different techniques. In the past years, we saw significant improvements in the field of sentence embeddings and especially towards the development of universal sentence encoders that could provide inductive transfer to a wide variety of downstream tasks. In this work, we perform a comprehensive evaluation of recent methods using a wide variety of downstream and linguistic feature probing tasks. We show that a simple approach using bag-of-words with a recently introduced language model for deep context-dependent word embeddings proved to yield better results in many tasks when compared to sentence encoders trained on entailment datasets. We also show, however, that we are still far away from a universal encoder that can perform consistently across several downstream tasks.