Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation
For researchers and practitioners using LLMs for translation, this work provides a practical alternative to internal confidence signals, though the methods are incremental and model-dependent.
The paper investigates the reliability of LLMs' verbalized confidence in machine translation, proposing five verbalized methods to extract per-token confidence without relying on internal signals. Results show that verbalized and internal methods perform similarly in error detection and calibration, but with little correlation between them.
The rapid rise in popularity of large language models (LLMs) for translation calls for a thorough study of the reliability of their confidence in their own outputs. Unlike many generation tasks, translation errors and confidence levels can be useful at different levels of granularity (tokens, words, or spans). Unsupervised approaches based on internal signals like predicted probabilities can be misleading because they reflect certainty among alternatives rather than correctness. In addition, they require access to such internal signals. Here, we devise five verbalized methods of extracting an LLM's per-token confidence without those shortcomings and compare their reliability with that of the model's internal signals of certainty. We evaluate reliability using two forms of alignment: fine-grained error detection and calibration. For both, internal and verbalized methods perform similarly, although results vary by model. Interestingly, we find little to no correlation between internal and verbalized methods.