Reframing Human-AI Collaboration for Generating Free-Text Explanations
This addresses the challenge of improving AI-generated explanations for human users, though it is incremental as it builds on existing few-shot methods and filtering techniques.
The paper tackled the problem of generating free-text explanations for classification decisions using large language models, finding that GPT-3-generated explanations are often preferred over crowdsourced ones, but still need improvement in aspects like novelty and label support, and developed a pipeline with a supervised filter to enhance acceptability.
Large language models are increasingly capable of generating fluent-appearing text with relatively little task-specific supervision. But can these models accurately explain classification decisions? We consider the task of generating free-text explanations using human-written examples in a few-shot manner. We find that (1) authoring higher quality prompts results in higher quality generations; and (2) surprisingly, in a head-to-head comparison, crowdworkers often prefer explanations generated by GPT-3 to crowdsourced explanations in existing datasets. Our human studies also show, however, that while models often produce factual, grammatical, and sufficient explanations, they have room to improve along axes such as providing novel information and supporting the label. We create a pipeline that combines GPT-3 with a supervised filter that incorporates binary acceptability judgments from humans in the loop. Despite the intrinsic subjectivity of acceptability judgments, we demonstrate that acceptability is partially correlated with various fine-grained attributes of explanations. Our approach is able to consistently filter GPT-3-generated explanations deemed acceptable by humans.