Predicting generalization performance with correctness discriminators
This addresses the need for trustworthiness in NLP by enabling accuracy prediction for out-of-distribution data, though it is incremental as it builds on existing discriminator methods.
The paper tackled the problem of predicting NLP model accuracy on unseen data without gold labels by training a correctness discriminator to establish upper and lower bounds, showing that gold accuracy reliably falls within these bounds across multiple tasks.
The ability to predict an NLP model's accuracy on unseen, potentially out-of-distribution data is a prerequisite for trustworthiness. We present a novel model that establishes upper and lower bounds on the accuracy, without requiring gold labels for the unseen data. We achieve this by training a discriminator which predicts whether the output of a given sequence-to-sequence model is correct or not. We show across a variety of tagging, parsing, and semantic parsing tasks that the gold accuracy is reliably between the predicted upper and lower bounds, and that these bounds are remarkably close together.