Beyond Black-Box Labels: Interpretable Criteria for Diagnosing SubjectiveNLP Tasks
For NLP practitioners designing annotation schemas for subjective tasks, this diagnostic offers a method to detect and address sources of disagreement before committing to gold labels.
The paper proposes a schema-level diagnostic for auditing annotation schemas in subjective NLP tasks, using multi-annotator criterion judgments to identify unstable criteria and systematic category overlap. Applied to persuasive value extraction, it shows that disagreement is concentrated in a few criteria and nearly half of sentences activate multiple categories, providing evidence for improving annotation guidelines.
Subjective NLP datasets typically aggregate annotator judgments into a single gold label, making it difficult to diagnose whether disagreement reflects unclear criteria, collapsed distinctions, or legitimate plurality. We propose a \emph{schema-level diagnostic} for auditing expert-designed annotation schemas \emph{prior to} gold-label commitment, using only multi-annotator criterion judgments. The diagnostic separates two failure modes: unstable criteria with hard-to-operationalize boundaries, and systematic overlap that blurs the boundaries between mutually exclusive categories. Applied to persuasive value extraction in commercial documents, we find that disagreement is not diffuse: instability concentrates in a few criteria, while nearly half of covered sentences activate multiple categories. These signals align with where domain experts disagree, yielding an evidence-based audit for tightening guidelines, revising category structure, or reconsidering the annotation paradigm.