Joint Training for Selective Prediction
This work addresses the need for reliable human-in-the-loop systems in NLP by improving selective prediction methods, though it appears incremental as it builds on existing SP approaches.
The paper tackles the problem of selective prediction in NLP by introducing a joint-training approach that optimizes both classifier representations and a deferral policy, resulting in improved SP outcomes and enhanced performance of both modules across four classification tasks.
Classifier models are prevalent in natural language processing (NLP), often with high accuracy. Yet in real world settings, human-in-the-loop systems can foster trust in model outputs and even higher performance. Selective Prediction (SP) methods determine when to adopt a classifier's output versus defer to a human. Previous SP approaches have addressed how to improve softmax as a measure of model confidence, or have developed separate confidence estimators. One previous method involves learning a deferral model based on engineered features. We introduce a novel joint-training approach that simultaneously optimizes learned representations used by the classifier module and a learned deferral policy. Our results on four classification tasks demonstrate that joint training not only leads to better SP outcomes over two strong baselines, but also improves the performance of both modules.