Understanding Retrieval Augmentation for Long-Form Question Answering
This addresses the problem of understanding and improving attribution in retrieval-augmented generation for long-form QA, which is incremental as it builds on existing methods with controlled studies.
The paper investigates how retrieval-augmented language models use documents for long-form question answering, finding that generated answers are only partially attributable to the documents, especially for models not trained with retrieval augmentation.
How retrieved documents are used in language models (LMs) for long-form generation task is understudied. We present two controlled studies on retrieval-augmented LM for long-form question answering (LFQA): one fixing the LM and varying evidence documents and the other fixing evidence documents and varying the LMs. We study various attributes of generated answers (e.g., fluency, length, variance), with an emphasis on the attribution of generated answers to in-context evidence documents. We collect a dataset (SALAD) containing human annotations of sentence-level answer attribution in LFQA and evaluate existing methods for automatically judging attribution. We find that while LMs can leverage relevant in-context documents, the generated answer is only partially attributable towards the documents, especially for LMs trained without retrieval augmentation. Together, our analysis reveals how retrieval augmentation impacts long knowledge-rich text generation and provide directions for future work.