A Survey of Pre-trained Language Models for Processing Scientific Text
It helps researchers by organizing and comparing the growing number of scientific language models, but it is incremental as it is a survey rather than new research.
This paper addresses the lack of comprehensive surveys on pre-trained language models for scientific text by providing a review and analysis of their effectiveness across domains, tasks, and datasets.
The number of Language Models (LMs) dedicated to processing scientific text is on the rise. Keeping pace with the rapid growth of scientific LMs (SciLMs) has become a daunting task for researchers. To date, no comprehensive surveys on SciLMs have been undertaken, leaving this issue unaddressed. Given the constant stream of new SciLMs, appraising the state-of-the-art and how they compare to each other remain largely unknown. This work fills that gap and provides a comprehensive review of SciLMs, including an extensive analysis of their effectiveness across different domains, tasks and datasets, and a discussion on the challenges that lie ahead.