Learning to Reject with a Fixed Predictor: Application to Decontextualization
This work addresses the challenge of improving reliability in NLP tasks like decontextualization by enabling fixed predictors to reject uncertain predictions, though it is incremental as it builds on existing rejection frameworks.
The paper tackles the problem of classification with a reject option for a fixed predictor in NLP, introducing a new formulation and algorithm that achieves a ~25% improvement in coverage when halving the error rate, only ~3% away from the theoretical limit.
We study the problem of classification with a reject option for a fixed predictor, applicable in natural language processing. We introduce a new problem formulation for this scenario, and an algorithm minimizing a new surrogate loss function. We provide a complete theoretical analysis of the surrogate loss function with a strong $H$-consistency guarantee. For evaluation, we choose the decontextualization task, and provide a manually-labelled dataset of $2\mathord,000$ examples. Our algorithm significantly outperforms the baselines considered, with a $\sim\!\!25\%$ improvement in coverage when halving the error rate, which is only $\sim\!\! 3 \%$ away from the theoretical limit.