Designing NLP Systems That Adapt to Diverse Worldviews
This addresses the plateau in NLI progress for AI language understanding by incorporating diverse worldviews, though it is incremental as it builds on existing datasets like SBIC.
The paper tackles the problem of Natural Language Inference (NLI) models failing on ambiguous examples and poor generalization by arguing that this stems from ignoring the subjective nature of meaning tied to worldviews. It proposes a perspectivist approach with datasets capturing annotator demographics and justifications, showing in initial experiments that limited annotator metadata can improve model performance.
Natural Language Inference (NLI) is foundational for evaluating language understanding in AI. However, progress has plateaued, with models failing on ambiguous examples and exhibiting poor generalization. We argue that this stems from disregarding the subjective nature of meaning, which is intrinsically tied to an individual's \textit{weltanschauung} (which roughly translates to worldview). Existing NLP datasets often obscure this by aggregating labels or filtering out disagreement. We propose a perspectivist approach: building datasets that capture annotator demographics, values, and justifications for their labels. Such datasets would explicitly model diverse worldviews. Our initial experiments with a subset of the SBIC dataset demonstrate that even limited annotator metadata can improve model performance.