Uncertain Natural Language Inference
This addresses the limitation of categorical bin assignment in NLI tasks for AI systems, though it is incremental as it refines an existing task rather than creating a new one.
The paper tackles the problem of Natural Language Inference (NLI) by shifting from categorical labels to predicting subjective probability assessments, introducing Uncertain Natural Language Inference (UNLI). They demonstrate feasibility by relabeling part of the SNLI dataset and show that their best models approach human performance.
We introduce Uncertain Natural Language Inference (UNLI), a refinement of Natural Language Inference (NLI) that shifts away from categorical labels, targeting instead the direct prediction of subjective probability assessments. We demonstrate the feasibility of collecting annotations for UNLI by relabeling a portion of the SNLI dataset under a probabilistic scale, where items even with the same categorical label differ in how likely people judge them to be true given a premise. We describe a direct scalar regression modeling approach, and find that existing categorically labeled NLI data can be used in pre-training. Our best models approach human performance, demonstrating models may be capable of more subtle inferences than the categorical bin assignment employed in current NLI tasks.