Retrieval-Augmented Linguistic Calibration
For practitioners needing reliable confidence expressions from LLMs, RALC offers a principled, generalizable calibration framework that handles linguistic cues and audience interpretation variability.
The paper introduces Retrieval-Augmented Linguistic Calibration (RALC), a post-hoc pipeline that improves linguistic confidence calibration in LLMs by propagating calibrated signals into natural language via retrieval-augmented rewriting. Across three QA benchmarks and five LLM families, RALC achieves up to 66% improvement in faithfulness and 58% in calibration, outperforming baselines.
Linguistic cues such as "I believe" and "probably" offer an intuitive interface for communicating confidence, yet a generalisable, principled calibration framework for linguistic confidence expressions remains underexplored. In particular, co-occurring linguistic cues, contextual variation, and subjective audience interpretation pose unique challenges. We therefore model linguistic confidence as a distribution over plausible perceived probability values that a statement is correct, capturing interpretation variability that scalar representations discard. Within this distributional framework, we introduce faithfulness as a complementary evaluation dimension and present Faithfulness Divergence (FD), an information-theoretic metric quantifying the surprise induced in audience beliefs upon truth revelation. Building on these foundations, we present Retrieval-Augmented Linguistic Calibration (RALC), a lightweight post-hoc pipeline that propagates calibrated confidence signals back into natural language via retrieval-augmented rewriting. Across three QA benchmarks and five LLM families, RALC improves in-domain faithfulness and calibration up to 66% and 58%, respectively, outperforming black-box and grey-box calibration baselines.