How can LLMs Support Policy Researchers? Evaluating an LLM-Assisted Workflow for Large-Scale Unstructured Data
This addresses the problem of slow, expensive, and limited-scale thematic analysis for policy researchers, though it is incremental as it builds on existing workflows.
The study tackled the challenge of scaling thematic analysis for policy research by evaluating an LLM-assisted workflow on large-scale unstructured data, such as millions of Reddit posts and over 1,000 chatbot-led interview transcripts, and found it provided quick, rough-and-ready insights that aligned with and diverged from authoritative reports.
Policy researchers need scalable ways to surface public views, yet they often rely on interviews, listening sessions, and surveys-analyzed thematically-that are slow, expensive, and limited in scale and diversity. LLMs offer new possibilities for thematic analysis of unstructured text, yet we know little about how LLM-assisted workflows perform for policy research. Building on a workflow for LLM-assisted thematic analysis of online forums, we conduct a study with 11 policy researchers, who use an early prototype and see it as a quick, rough-and-ready input to their research. We then extend and scale the workflow to analyze millions of Reddit posts and 1,058 chatbot-led interview transcripts on a policy-relevant topic, treating these sources as rich and scalable data for policy discourse. We compare the synthesized themes to those from authoritative policy reports, identify points of alignment and divergence, and discuss what this implies for policy researchers adopting LLM-assisted workflows.