Conformal Linguistic Calibration: Trading-off between Factuality and Specificity
This addresses the issue of leveraging uncertain model responses in downstream tasks for users of language models, though it appears incremental as it builds on existing approaches.
The paper tackles the problem of unreliable language model outputs by proposing Conformal Linguistic Calibration (CLC), which unifies abstention and linguistic calibration into answer set prediction, resulting in calibrated outputs with conformal guarantees on factual accuracy and enabling a controllable balance between factuality and specificity.
Language model outputs are not always reliable, thus prompting research into how to adapt model responses based on uncertainty. Common approaches include: \emph{abstention}, where models refrain from generating responses when uncertain; and \emph{linguistic calibration}, where models hedge their statements using uncertainty quantifiers. However, abstention can withhold valuable information, while linguistically calibrated responses are often challenging to leverage in downstream tasks. We propose a unified view, Conformal Linguistic Calibration (CLC), which reinterprets linguistic calibration as \emph{answer set prediction}. First we present a framework connecting abstention and linguistic calibration through the lens of linguistic pragmatics. We then describe an implementation of CLC that allows for controlling the level of imprecision in model responses. Results demonstrate our method produces calibrated outputs with conformal guarantees on factual accuracy. Further, our approach enables fine-tuning models to perform uncertainty-aware adaptive claim rewriting, offering a controllable balance between factuality and specificity.