Can Unconfident LLM Annotations Be Used for Confident Conclusions?
This addresses the challenge of high costs and slow human annotation in computational social science and NLP, offering a method to efficiently combine LLM and human inputs, though it is incremental as it builds on existing LLM annotation practices.
The paper tackles the problem of using LLM annotations to reduce the need for expensive human annotations in computational social science, introducing Confidence-Driven Inference to strategically select human annotations based on LLM confidence, which reduces required human annotations by over 25% while ensuring valid conclusions.
Large language models (LLMs) have shown high agreement with human raters across a variety of tasks, demonstrating potential to ease the challenges of human data collection. In computational social science (CSS), researchers are increasingly leveraging LLM annotations to complement slow and expensive human annotations. Still, guidelines for collecting and using LLM annotations, without compromising the validity of downstream conclusions, remain limited. We introduce Confidence-Driven Inference: a method that combines LLM annotations and LLM confidence indicators to strategically select which human annotations should be collected, with the goal of producing accurate statistical estimates and provably valid confidence intervals while reducing the number of human annotations needed. Our approach comes with safeguards against LLM annotations of poor quality, guaranteeing that the conclusions will be both valid and no less accurate than if we only relied on human annotations. We demonstrate the effectiveness of Confidence-Driven Inference over baselines in statistical estimation tasks across three CSS settings--text politeness, stance, and bias--reducing the needed number of human annotations by over 25% in each. Although we use CSS settings for demonstration, Confidence-Driven Inference can be used to estimate most standard quantities across a broad range of NLP problems.