LeWiDi-2025 at NLPerspectives: The Third Edition of the Learning with Disagreements Shared Task
This work addresses the need for AI models to handle disagreement in human judgments, offering resources and benchmarks for researchers in natural language processing, though it is incremental as it builds on previous editions.
The paper presents the third edition of the Learning with Disagreements (LeWiDi) shared task, which extended the benchmark to four datasets with categorical and ordinal judgments, and introduced two evaluation paradigms (soft-label and perspectivist) with new metrics. The task attracted diverse participation and provided insights into methods for modeling variation in human judgments.
Many researchers have reached the conclusion that AI models should be trained to be aware of the possibility of variation and disagreement in human judgments, and evaluated as per their ability to recognize such variation. The LEWIDI series of shared tasks on Learning With Disagreements was established to promote this approach to training and evaluating AI models, by making suitable datasets more accessible and by developing evaluation methods. The third edition of the task builds on this goal by extending the LEWIDI benchmark to four datasets spanning paraphrase identification, irony detection, sarcasm detection, and natural language inference, with labeling schemes that include not only categorical judgments as in previous editions, but ordinal judgments as well. Another novelty is that we adopt two complementary paradigms to evaluate disagreement-aware systems: the soft-label approach, in which models predict population-level distributions of judgments, and the perspectivist approach, in which models predict the interpretations of individual annotators. Crucially, we moved beyond standard metrics such as cross-entropy, and tested new evaluation metrics for the two paradigms. The task attracted diverse participation, and the results provide insights into the strengths and limitations of methods to modeling variation. Together, these contributions strengthen LEWIDI as a framework and provide new resources, benchmarks, and findings to support the development of disagreement-aware technologies.