Paired Completion: Flexible Quantification of Issue-framing at Scale with LLMs
This provides a scalable tool for social science and policy analysis to analyze issue framing in text, though it is incremental as it builds on existing LLM methods.
The paper tackled the problem of detecting issue framing in text, which is challenging due to subtle linguistic differences, by introducing 'paired completion', a novel approach using LLM next-token log probabilities with minimal examples. The result showed it is a cost-efficient, low-bias alternative to existing methods, offering a scalable solution for large text collections, especially in low-resource settings.
Detecting issue framing in text - how different perspectives approach the same topic - is valuable for social science and policy analysis, yet challenging for automated methods due to subtle linguistic differences. We introduce `paired completion', a novel approach using LLM next-token log probabilities to detect contrasting frames using minimal examples. Through extensive evaluation across synthetic datasets and a human-labeled corpus, we demonstrate that paired completion is a cost-efficient, low-bias alternative to both prompt-based and embedding-based methods, offering a scalable solution for analyzing issue framing in large text collections, especially suited to low-resource settings.