The Reliability Paradox: Exploring How Shortcut Learning Undermines Language Model Calibration
This work addresses a critical issue in natural language processing by revealing that current calibration metrics may mislead about model reliability, which is important for researchers and practitioners developing robust AI systems.
The paper investigates the relationship between language model calibration and shortcut learning, finding that models with lower calibration error often rely more on non-generalizable decision rules, challenging the assumption that well-calibrated models are reliable.
The advent of pre-trained language models (PLMs) has enabled significant performance gains in the field of natural language processing. However, recent studies have found PLMs to suffer from miscalibration, indicating a lack of accuracy in the confidence estimates provided by these models. Current evaluation methods for PLM calibration often assume that lower calibration error estimates indicate more reliable predictions. However, fine-tuned PLMs often resort to shortcuts, leading to overconfident predictions that create the illusion of enhanced performance but lack generalizability in their decision rules. The relationship between PLM reliability, as measured by calibration error, and shortcut learning, has not been thoroughly explored thus far. This paper aims to investigate this relationship, studying whether lower calibration error implies reliable decision rules for a language model. Our findings reveal that models with seemingly superior calibration portray higher levels of non-generalizable decision rules. This challenges the prevailing notion that well-calibrated models are inherently reliable. Our study highlights the need to bridge the current gap between language model calibration and generalization objectives, urging the development of comprehensive frameworks to achieve truly robust and reliable language models.