Fine-Grained Detection of Solidarity for Women and Migrants in 155 Years of German Parliamentary Debates
This work addresses the challenge of analyzing social relations in historical texts for social scientists, offering a cost-effective LLM-based method, though it is incremental as it applies existing models to a new dataset.
The paper tackled the problem of detecting fine-grained solidarity frames towards women and migrants in German parliamentary debates over 155 years, finding that GPT-4 outperformed other LLMs in annotation quality and revealed shifts in solidarity types, such as a decline in group-based notions in favor of compassionate and exchange-based solidarity.
Solidarity is a crucial concept to understand social relations in societies. In this paper, we explore fine-grained solidarity frames to study solidarity towards women and migrants in German parliamentary debates between 1867 and 2022. Using 2,864 manually annotated text snippets (with a cost exceeding 18k Euro), we evaluate large language models (LLMs) like Llama 3, GPT-3.5, and GPT-4. We find that GPT-4 outperforms other LLMs, approaching human annotation quality. Using GPT-4, we automatically annotate more than 18k further instances (with a cost of around 500 Euro) across 155 years and find that solidarity with migrants outweighs anti-solidarity but that frequencies and solidarity types shift over time. Most importantly, group-based notions of (anti-)solidarity fade in favor of compassionate solidarity, focusing on the vulnerability of migrant groups, and exchange-based anti-solidarity, focusing on the lack of (economic) contribution. Our study highlights the interplay of historical events, socio-economic needs, and political ideologies in shaping migration discourse and social cohesion. We also show that powerful LLMs, if carefully prompted, can be cost-effective alternatives to human annotation for hard social scientific tasks.