Language Models with Conformal Factuality Guarantees
This addresses the open problem of guaranteeing factuality in language models, which is crucial for reliable AI applications, though it is an incremental improvement based on existing conformal prediction methods.
The paper tackles the problem of ensuring the correctness and factuality of language model outputs by proposing a framework called conformal factuality, which uses conformal prediction to provide high-probability correctness guarantees, achieving 80-90% correctness while retaining most of the original output in evaluations on QA and reasoning tasks.
Guaranteeing the correctness and factuality of language model (LM) outputs is a major open problem. In this work, we propose conformal factuality, a framework that can ensure high probability correctness guarantees for LMs by connecting language modeling and conformal prediction. We observe that the correctness of an LM output is equivalent to an uncertainty quantification problem, where the uncertainty sets are defined as the entailment set of an LM's output. Using this connection, we show that conformal prediction in language models corresponds to a back-off algorithm that provides high probability correctness guarantees by progressively making LM outputs less specific (and expanding the associated uncertainty sets). This approach applies to any black-box LM and requires very few human-annotated samples. Evaluations of our approach on closed book QA (FActScore, NaturalQuestions) and reasoning tasks (MATH) show that our approach can provide 80-90% correctness guarantees while retaining the majority of the LM's original output.