Transformer Encoder for Social Science
This addresses the challenge for social scientists who need efficient text analysis with small datasets, though it is incremental as it adapts existing methods to a specific domain.
The paper tackles the problem of limited training data in social science text processing by proposing TESS, a compact pretrained transformer encoder, which outperforms BERT and RoBERTa by 16.7% on average with fewer than 1,000 training instances.
High-quality text data has become an important data source for social scientists. We have witnessed the success of pretrained deep neural network models, such as BERT and RoBERTa, in recent social science research. In this paper, we propose a compact pretrained deep neural network, Transformer Encoder for Social Science (TESS), explicitly designed to tackle text processing tasks in social science research. Using two validation tests, we demonstrate that TESS outperforms BERT and RoBERTa by 16.7% on average when the number of training samples is limited (<1,000 training instances). The results display the superiority of TESS over BERT and RoBERTa on social science text processing tasks. Lastly, we discuss the limitation of our model and present advice for future researchers.