Linguistic Knowledge and Transferability of Contextual Representations
This work provides insights into the transferability and limitations of contextual representations for NLP practitioners, though it is incremental in nature.
The study investigated the linguistic knowledge captured by contextual word representations from models like ELMo, OpenAI transformer, and BERT using 17 probing tasks, finding that linear models on frozen representations are competitive with SOTA in many cases but fail on fine-grained tasks like conjunct identification. It also analyzed transferability, showing that higher RNN layers are more task-specific, while transformers do not follow this trend, and that supervised pretraining on related tasks outperforms language model pretraining with fixed data, but language models with more data yield the best results.
Contextual word representations derived from large-scale neural language models are successful across a diverse set of NLP tasks, suggesting that they encode useful and transferable features of language. To shed light on the linguistic knowledge they capture, we study the representations produced by several recent pretrained contextualizers (variants of ELMo, the OpenAI transformer language model, and BERT) with a suite of seventeen diverse probing tasks. We find that linear models trained on top of frozen contextual representations are competitive with state-of-the-art task-specific models in many cases, but fail on tasks requiring fine-grained linguistic knowledge (e.g., conjunct identification). To investigate the transferability of contextual word representations, we quantify differences in the transferability of individual layers within contextualizers, especially between recurrent neural networks (RNNs) and transformers. For instance, higher layers of RNNs are more task-specific, while transformer layers do not exhibit the same monotonic trend. In addition, to better understand what makes contextual word representations transferable, we compare language model pretraining with eleven supervised pretraining tasks. For any given task, pretraining on a closely related task yields better performance than language model pretraining (which is better on average) when the pretraining dataset is fixed. However, language model pretraining on more data gives the best results.