EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI
This work addresses the challenge of scaling error detection for NLP datasets, reducing human effort while enhancing dataset quality, though it is incremental as it builds on prior manual frameworks.
The study tackled the problem of detecting annotation errors in natural language inference datasets under human label variation by proposing EVADE, a framework using large language models to generate and validate explanations, resulting in improved alignment with human annotations and better fine-tuning performance when removing LLM-detected errors compared to human-detected ones.
High-quality datasets are critical for training and evaluating reliable NLP models. In tasks like natural language inference (NLI), human label variation (HLV) arises when multiple labels are valid for the same instance, making it difficult to separate annotation errors from plausible variation. An earlier framework VARIERR (Weber-Genzel et al., 2024) asks multiple annotators to explain their label decisions in the first round and flag errors via validity judgments in the second round. However, conducting two rounds of manual annotation is costly and may limit the coverage of plausible labels or explanations. Our study proposes a new framework, EVADE, for generating and validating explanations to detect errors using large language models (LLMs). We perform a comprehensive analysis comparing human- and LLM-detected errors for NLI across distribution comparison, validation overlap, and impact on model fine-tuning. Our experiments demonstrate that LLM validation refines generated explanation distributions to more closely align with human annotations, and that removing LLM-detected errors from training data yields improvements in fine-tuning performance than removing errors identified by human annotators. This highlights the potential to scale error detection, reducing human effort while improving dataset quality under label variation.