On the Calibration of Massively Multilingual Language Models
This addresses the problem of unreliable confidence estimates in MMLMs for users in multilingual NLP applications, though it is incremental as it builds on existing calibration techniques.
The paper investigates the calibration of Massively Multilingual Language Models (MMLMs), finding they are miscalibrated, especially for low-resource or typologically diverse languages, and shows that methods like temperature scaling and few-shot examples can reduce calibration errors.
Massively Multilingual Language Models (MMLMs) have recently gained popularity due to their surprising effectiveness in cross-lingual transfer. While there has been much work in evaluating these models for their performance on a variety of tasks and languages, little attention has been paid on how well calibrated these models are with respect to the confidence in their predictions. We first investigate the calibration of MMLMs in the zero-shot setting and observe a clear case of miscalibration in low-resource languages or those which are typologically diverse from English. Next, we empirically show that calibration methods like temperature scaling and label smoothing do reasonably well towards improving calibration in the zero-shot scenario. We also find that few-shot examples in the language can further help reduce the calibration errors, often substantially. Overall, our work contributes towards building more reliable multilingual models by highlighting the issue of their miscalibration, understanding what language and model specific factors influence it, and pointing out the strategies to improve the same.