Learning with Instance Bundles for Reading Comprehension
This addresses a bottleneck in reading comprehension training for NLP researchers, though it appears incremental as it builds on contrastive estimation ideas.
The paper tackles the problem of training reading comprehension models by leveraging relationships between related questions instead of treating them as independent, showing up to 11% absolute gains in accuracy on HotpotQA and ROPES datasets.
When training most modern reading comprehension models, all the questions associated with a context are treated as being independent from each other. However, closely related questions and their corresponding answers are not independent, and leveraging these relationships could provide a strong supervision signal to a model. Drawing on ideas from contrastive estimation, we introduce several new supervision techniques that compare question-answer scores across multiple related instances. Specifically, we normalize these scores across various neighborhoods of closely contrasting questions and/or answers, adding another cross entropy loss term that is used in addition to traditional maximum likelihood estimation. Our techniques require bundles of related question-answer pairs, which we can either mine from within existing data or create using various automated heuristics. We empirically demonstrate the effectiveness of training with instance bundles on two datasets -- HotpotQA and ROPES -- showing up to 11% absolute gains in accuracy.