Beyond the Link: Assessing LLMs' ability to Classify Political Content across Global Media
This addresses the need for cost-effective political content classification in digital media research, though it is incremental in improving existing methods.
The study evaluated whether large language models (LLMs) can accurately classify political content from URLs across five countries, finding that URLs provide a scalable alternative to full-text analysis but with systematic biases like overclassifying centrist news as political.
The use of large language models (LLMs) is becoming common in political science and digital media research. While LLMs have demonstrated ability in labelling tasks, their effectiveness to classify Political Content (PC) from URLs remains underexplored. This article evaluates whether LLMs can accurately distinguish PC from non-PC using both the text and the URLs of news articles across five countries (France, Germany, Spain, the UK, and the US) and their different languages. Using cutting-edge models, we benchmark their performance against human-coded data to assess whether URL-level analysis can approximate full-text analysis. Our findings show that URLs embed relevant information and can serve as a scalable, cost-effective alternative to discern PC. However, we also uncover systematic biases: LLMs seem to overclassify centrist news as political, leading to false positives that may distort further analyses. We conclude by outlining methodological recommendations on the use of LLMs in political science research.