Atomic Self-Consistency for Better Long Form Generations
This addresses the issue of incomplete information in long-form AI responses for users needing accurate and comprehensive answers, representing an incremental improvement over existing self-consistency techniques.
The paper tackles the problem of improving recall of relevant information in long-form LLM responses by introducing Atomic Self-Consistency (ASC), a technique that merges authentic subparts from multiple stochastic samples into a composite answer, showing significant gains over prior methods on multiple QA datasets.
Recent work has aimed to improve LLM generations by filtering out hallucinations, thereby improving the precision of the information in responses. Correctness of a long-form response, however, also depends on the recall of multiple pieces of information relevant to the question. In this paper, we introduce Atomic Self-Consistency (ASC), a technique for improving the recall of relevant information in an LLM response. ASC follows recent work, Universal Self-Consistency (USC) in using multiple stochastic samples from an LLM to improve the long-form response. Unlike USC which only focuses on selecting the best single generation, ASC picks authentic subparts from the samples and merges them into a superior composite answer. Through extensive experiments and ablations, we show that merging relevant subparts of multiple samples performs significantly better than picking a single sample. ASC demonstrates significant gains over USC on multiple factoids and open-ended QA datasets - ASQA, QAMPARI, QUEST, ELI5 with ChatGPT and Llama2. Our analysis also reveals untapped potential for enhancing long-form generations using approach of merging multiple samples.