Shared Doubt: Zero-shot Cross-Lingual Confidence Estimation for Language Models
This work addresses the problem of cross-lingual confidence estimation for large language models, which is crucial for deploying LLMs reliably in multilingual settings, especially when retraining for each language is not feasible.
This paper explores whether multilingual LLMs encode shared, language-transferable confidence features to enable zero-shot cross-lingual confidence estimation. They developed a lightweight linear probe that, when trained monolingually, generalizes zero-shot to typologically diverse languages, providing a strong baseline for cross-lingual confidence estimation.
Confidence estimation (CE), i.e. quantifying the reliability of a model's prediction, has attracted great interest in the context of large language models (LLMs). However, most studies focus on English, ignoring the multilingual reality of LLM usage, while many CE methods degrade or require retraining across languages. To address this gap, we investigate whether multilingual LLMs encode shared, language-transferable confidence features. We use a lightweight linear probe that predicts answer correctness directly from intermediate representations. Trained monolingually, the probe generalizes zero-shot to unseen, typologically diverse languages without target-language supervision. Learned layer weights and multiple ablations reveal that confidence features concentrate in middle layers across languages, suggesting a shared confidence subspace. While zero-shot cross-lingual performance depends on similarity to the source language, the probe provides a strong baseline without any retraining and compares favorably to other popular confidence estimation methods.