Pretrained Transformers Improve Out-of-Distribution Robustness
This work addresses the critical issue of model robustness for real-world NLP applications by providing systematic evidence that pretrained Transformers improve OOD generalization, though it is incremental as it builds on existing models and benchmarks.
The paper tackled the problem of out-of-distribution (OOD) generalization in NLP by constructing a new robustness benchmark with realistic distribution shifts across seven datasets, and found that pretrained Transformers like BERT show substantially smaller performance declines compared to previous models such as bag-of-words, ConvNets, and LSTMs, with many previous models performing worse than chance at detecting OOD examples.
Although pretrained Transformers such as BERT achieve high accuracy on in-distribution examples, do they generalize to new distributions? We systematically measure out-of-distribution (OOD) generalization for seven NLP datasets by constructing a new robustness benchmark with realistic distribution shifts. We measure the generalization of previous models including bag-of-words models, ConvNets, and LSTMs, and we show that pretrained Transformers' performance declines are substantially smaller. Pretrained transformers are also more effective at detecting anomalous or OOD examples, while many previous models are frequently worse than chance. We examine which factors affect robustness, finding that larger models are not necessarily more robust, distillation can be harmful, and more diverse pretraining data can enhance robustness. Finally, we show where future work can improve OOD robustness.