A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI
This work addresses the challenge of efficiently capturing human label variation in NLP tasks, offering a method to reduce the time-consuming process of collecting human explanations, though it is incremental as it builds on prior research using LLMs for HJDs.
The paper tackles the problem of approximating human judgment distributions (HJDs) in natural language inference (NLI) by using LLM-generated explanations as proxies for human explanations, showing that they yield comparable results to human explanations when provided with human labels, with generalization to datasets lacking human explanations and out-of-distribution test sets.
Disagreement in human labeling is ubiquitous, and can be captured in human judgment distributions (HJDs). Recent research has shown that explanations provide valuable information for understanding human label variation (HLV) and large language models (LLMs) can approximate HJD from a few human-provided label-explanation pairs. However, collecting explanations for every label is still time-consuming. This paper examines whether LLMs can be used to replace humans in generating explanations for approximating HJD. Specifically, we use LLMs as annotators to generate model explanations for a few given human labels. We test ways to obtain and combine these label-explanations with the goal to approximate human judgment distributions. We further compare the resulting human with model-generated explanations, and test automatic and human explanation selection. Our experiments show that LLM explanations are promising for NLI: to estimate HJDs, generated explanations yield comparable results to human's when provided with human labels. Importantly, our results generalize from datasets with human explanations to i) datasets where they are not available and ii) challenging out-of-distribution test sets.