Arti-"fickle" Intelligence: Using LLMs as a Tool for Inference in the Political and Social Sciences
It addresses the problem of ensuring reliable scientific inference with LLMs for researchers in political and social sciences, but is incremental as it focuses on guidelines rather than new methods.
The paper tackles the challenge of using large language models (LLMs) for scientific inference in political and social sciences, proposing guidelines for validating model outputs to improve shared knowledge.
Generative large language models (LLMs) are incredibly useful, versatile, and promising tools. However, they will be of most use to political and social science researchers when they are used in a way that advances understanding about real human behaviors and concerns. To promote the scientific use of LLMs, we suggest that researchers in the political and social sciences need to remain focused on the scientific goal of inference. To this end, we discuss the challenges and opportunities related to scientific inference with LLMs, using validation of model output as an illustrative case for discussion. We propose a set of guidelines related to establishing the failure and success of LLMs when completing particular tasks, and discuss how we can make inferences from these observations. We conclude with a discussion of how this refocus will improve the accumulation of shared scientific knowledge about these tools and their uses in the social sciences.