Contextual Embeddings: When Are They Worth It?
This work addresses the practical problem for NLP practitioners of determining when to invest in computationally expensive contextual embeddings versus using simpler alternatives, revealing that gains are often incremental and task-dependent.
The paper investigates when deep contextual embeddings like BERT outperform simpler baselines such as GloVe or random embeddings, finding that these baselines can match or come close to contextual embeddings on large-scale data, often within 5-10% accuracy on benchmarks, with contextual embeddings showing significant gains only for tasks involving complex structure, ambiguity, or unseen words.
We study the settings for which deep contextual embeddings (e.g., BERT) give large improvements in performance relative to classic pretrained embeddings (e.g., GloVe), and an even simpler baseline---random word embeddings---focusing on the impact of the training set size and the linguistic properties of the task. Surprisingly, we find that both of these simpler baselines can match contextual embeddings on industry-scale data, and often perform within 5 to 10% accuracy (absolute) on benchmark tasks. Furthermore, we identify properties of data for which contextual embeddings give particularly large gains: language containing complex structure, ambiguous word usage, and words unseen in training.