Combining Autoregressive and Autoencoder Language Models for Text Classification
This provides a scalable solution for automated content analysis in social science research, minimizing sample size requirements, but it is incremental as it combines existing model types.
The paper tackles the problem of text classification by integrating autoregressive and autoencoder language models, resulting in a method that consistently outperforms existing methods on benchmark datasets, particularly in tasks with smaller datasets and more abstract objectives.
This paper presents CAALM-TC (Combining Autoregressive and Autoencoder Language Models for Text Classification), a novel method that enhances text classification by integrating autoregressive and autoencoder language models. Autoregressive large language models such as Open AI's GPT, Meta's Llama or Microsoft's Phi offer promising prospects for content analysis practitioners, but they generally underperform supervised BERT based models for text classification. CAALM leverages autoregressive models to generate contextual information based on input texts, which is then combined with the original text and fed into an autoencoder model for classification. This hybrid approach capitalizes on the extensive contextual knowledge of autoregressive models and the efficient classification capabilities of autoencoders. Experimental results on four benchmark datasets demonstrate that CAALM consistently outperforms existing methods, particularly in tasks with smaller datasets and more abstract classification objectives. The findings indicate that CAALM offers a scalable and effective solution for automated content analysis in social science research that minimizes sample size requirements.