Populism Meets AI: Advancing Populism Research with LLMs
This provides a scalable, cost-effective tool for political scientists and researchers analyzing populism across languages and large corpora, though it is incremental as it adapts existing methods to a specific domain.
The researchers tackled the challenge of measuring populism's ideational content by using a rubric and anchor guided chain of thought prompting with LLMs, achieving classification accuracy on par with expert human coders.
Measuring the ideational content of populism remains a challenge. Traditional strategies based on textual analysis have been critical for building the field's foundations and providing a valid, objective indicator of populist framing. Yet these approaches are costly, time consuming, and difficult to scale across languages, contexts, and large corpora. Here we present the results from a rubric and anchor guided chain of thought (CoT) prompting approach that mirrors human coder training. By leveraging the Global Populism Database (GPD), a comprehensive dataset of global leaders' speeches annotated for degrees of populism, we replicate the process used to train human coders by prompting the LLM with an adapted version of the same documentation to guide the model's reasoning. We then test multiple proprietary and open weight models by replicating scores in the GPD. Our findings reveal that this domain specific prompting strategy enables the LLM to achieve classification accuracy on par with expert human coders, demonstrating its ability to navigate the nuanced, context sensitive aspects of populism.