Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionality
This addresses the problem of data scarcity in human-level NLP tasks like mental health prediction for researchers, but it is incremental as it builds on existing methods.
The study tackled the challenge of using large transformer models for human-level NLP tasks with limited data by evaluating dimension reduction methods and found that fine-tuning is difficult but can be improved with pre-trained reduction, with RoBERTa performing best and PCA being effective for longer texts, achieving comparable results with only 1/12 of the embedding dimensions.
In human-level NLP tasks, such as predicting mental health, personality, or demographics, the number of observations is often smaller than the standard 768+ hidden state sizes of each layer within modern transformer-based language models, limiting the ability to effectively leverage transformers. Here, we provide a systematic study on the role of dimension reduction methods (principal components analysis, factorization techniques, or multi-layer auto-encoders) as well as the dimensionality of embedding vectors and sample sizes as a function of predictive performance. We first find that fine-tuning large models with a limited amount of data pose a significant difficulty which can be overcome with a pre-trained dimension reduction regime. RoBERTa consistently achieves top performance in human-level tasks, with PCA giving benefit over other reduction methods in better handling users that write longer texts. Finally, we observe that a majority of the tasks achieve results comparable to the best performance with just $\frac{1}{12}$ of the embedding dimensions.