From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives?
For researchers and practitioners needing to estimate collective human perspectives, this paper provides a principled framework for when LLMs can serve as superior estimators, repositioning them from a cost-saving compromise to a frontline tool.
This work challenges the view that LLMs are merely fallback annotators, showing they can outperform human annotators (including in-group humans) in predicting aggregate subgroup opinions on subjective tasks due to low variance and reduced bias coupling, with conditions common in practice.
Although large language models (LLMs) are increasingly used as annotators at scale, they are typically treated as a pragmatic fallback rather than a faithful estimator of human perspectives. This work challenges that presumption. By framing perspective-taking as the estimation of a latent group-level judgment, we characterize the conditions under which modern LLMs can outperform human annotators, including in-group humans, when predicting aggregate subgroup opinions on subjective tasks, and show that these conditions are common in practice. This advantage arises from structural properties of LLMs as estimators, including low variance and reduced coupling between representation and processing biases, rather than any claim of lived experience. Our analysis identifies clear regimes where LLMs act as statistically superior frontline estimators, as well as principled limits where human judgment remains essential. These findings reposition LLMs from a cost-saving compromise to a principled tool for estimating collective human perspectives.