Which *BERT? A Survey Organizing Contextualized Encoders
This provides a systematic overview for NLP researchers and practitioners to navigate and choose among diverse encoder models, though it is incremental as it synthesizes existing work rather than introducing new methods.
The paper tackles the problem of organizing the rapidly growing landscape of pretrained contextualized text encoders in NLP by presenting a survey that consolidates lessons learned and categorizes advances into common themes, with the result being a structured framework to guide model selection and interpretation of contributions.
Pretrained contextualized text encoders are now a staple of the NLP community. We present a survey on language representation learning with the aim of consolidating a series of shared lessons learned across a variety of recent efforts. While significant advancements continue at a rapid pace, we find that enough has now been discovered, in different directions, that we can begin to organize advances according to common themes. Through this organization, we highlight important considerations when interpreting recent contributions and choosing which model to use.