Introduction to Neural Transfer Learning with Transformers for Social Science Text Analysis
This work aims to enable social scientists with limited annotation resources to achieve higher prediction accuracies on text-based supervised learning tasks by leveraging Transformer models.
This paper introduces Transformer-based transfer learning to social scientists for text analysis, explaining its mechanisms, advantages, and limitations. It compares BERT, RoBERTa, and Longformer against conventional machine learning algorithms across three applications, demonstrating that Transformers consistently outperform traditional methods regardless of textual style or training data size.
Transformer-based models for transfer learning have the potential to achieve high prediction accuracies on text-based supervised learning tasks with relatively few training data instances. These models are thus likely to benefit social scientists that seek to have as accurate as possible text-based measures but only have limited resources for annotating training data. To enable social scientists to leverage these potential benefits for their research, this paper explains how these methods work, why they might be advantageous, and what their limitations are. Additionally, three Transformer-based models for transfer learning, BERT (Devlin et al. 2019), RoBERTa (Liu et al. 2019), and the Longformer (Beltagy et al. 2020), are compared to conventional machine learning algorithms on three applications. Across all evaluated tasks, textual styles, and training data set sizes, the conventional models are consistently outperformed by transfer learning with Transformers, thereby demonstrating the benefits these models can bring to text-based social science research.