Human-LLM Hybrid Text Answer Aggregation for Crowd Annotations
This addresses the challenge of improving annotation quality in crowdsourcing by leveraging LLMs as aggregators, though it is incremental as it builds on existing aggregation methods.
The paper tackles the problem of aggregating close-ended text answers from crowd workers by proposing a human-LLM hybrid method within a Creator-Aggregator Multi-Stage framework, showing effectiveness in experiments on public datasets.
The quality is a crucial issue for crowd annotations. Answer aggregation is an important type of solution. The aggregated answers estimated from multiple crowd answers to the same instance are the eventually collected annotations, rather than the individual crowd answers themselves. Recently, the capability of Large Language Models (LLMs) on data annotation tasks has attracted interest from researchers. Most of the existing studies mainly focus on the average performance of individual crowd workers; several recent works studied the scenarios of aggregation on categorical labels and LLMs used as label creators. However, the scenario of aggregation on text answers and the role of LLMs as aggregators are not yet well-studied. In this paper, we investigate the capability of LLMs as aggregators in the scenario of close-ended crowd text answer aggregation. We propose a human-LLM hybrid text answer aggregation method with a Creator-Aggregator Multi-Stage (CAMS) crowdsourcing framework. We make the experiments based on public crowdsourcing datasets. The results show the effectiveness of our approach based on the collaboration of crowd workers and LLMs.