Capturing Label Distribution: A Case Study in NLI
This work addresses annotation variability in NLP for researchers, but it is incremental as it builds on existing methods for label distribution estimation.
The paper tackles the problem of estimating human disagreement in natural language inference by comparing post-hoc smoothing and training with multiple references, finding that smoothing reduces KL divergence by almost half but does not improve accuracy, while multiple references enhance accuracy under a fixed annotation budget.
We study estimating inherent human disagreement (annotation label distribution) in natural language inference task. Post-hoc smoothing of the predicted label distribution to match the expected label entropy is very effective. Such simple manipulation can reduce KL divergence by almost half, yet will not improve majority label prediction accuracy or learn label distributions. To this end, we introduce a small amount of examples with multiple references into training. We depart from the standard practice of collecting a single reference per each training example, and find that collecting multiple references can achieve better accuracy under the fixed annotation budget. Lastly, we provide rich analyses comparing these two methods for improving label distribution estimation.