Calibration of Natural Language Understanding Models with Venn--ABERS Predictors
This addresses calibration issues for users relying on transformer models in NLU tasks, but it is incremental as it applies an existing calibration method to new models.
The paper tackled the problem of uncalibrated predictions in transformer-based natural language understanding models by proposing inductive Venn--ABERS predictors, which achieved well-calibrated probabilistic predictions uniformly spread over [0,1] while retaining the original model's accuracy.
Transformers, currently the state-of-the-art in natural language understanding (NLU) tasks, are prone to generate uncalibrated predictions or extreme probabilities, making the process of taking different decisions based on their output relatively difficult. In this paper we propose to build several inductive Venn--ABERS predictors (IVAP), which are guaranteed to be well calibrated under minimal assumptions, based on a selection of pre-trained transformers. We test their performance over a set of diverse NLU tasks and show that they are capable of producing well-calibrated probabilistic predictions that are uniformly spread over the [0,1] interval -- all while retaining the original model's predictive accuracy.