Conformal Autoregressive Generation: Beam Search with Coverage Guarantees
This addresses the need for uncertainty quantification in sequence generation for applications like NLP and chemistry, though it is incremental as it builds on existing beam search and conformal prediction methods.
The paper tackles the problem of generating reliable sequence sets by extending beam search with conformal prediction to provide theoretical coverage guarantees, achieving coverage rates of 92-98% on NLP and chemistry tasks.
We introduce two new extensions to the beam search algorithm based on conformal predictions (CP) to produce sets of sequences with theoretical coverage guarantees. The first method is very simple and proposes dynamically-sized subsets of beam search results but, unlike typical CP procedures, has an upper bound on the achievable guarantee depending on a post-hoc calibration measure. Our second algorithm introduces the conformal set prediction procedure as part of the decoding process, producing a variable beam width which adapts to the current uncertainty. While more complex, this procedure can achieve coverage guarantees selected a priori. We provide marginal coverage bounds for each method, and evaluate them empirically on a selection of tasks drawing from natural language processing and chemistry.