Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation
This addresses the issue of document order sensitivity in retrieval-augmented generation for question answering, though it appears incremental as it builds on known phenomena with a new gauge.
The paper tackles the problem that retrieval-augmented generation models are sensitive to document order in question answering, proposing pointwise mutual information (PMI) as a performance gauge that doesn't require prior knowledge of answers. They show PMI correlates with answer accuracy and use it to improve prompt selection and construction, achieving better performance in experiments on two QA datasets with various large language models.
Recent work suggests that large language models enhanced with retrieval-augmented generation are easily influenced by the order, in which the retrieved documents are presented to the model when solving tasks such as question answering (QA). However, there is no method to date that exploits this phenomenon to improve generation. We fill this gap. In this study, we show that the pointwise mutual information between a context and a question is an effective gauge for language model performance. Importantly, this gauge does not depend on knowing the answer to the question a priori. Through experiments on two question-answering datasets and a variety of large language models, we find evidence for an empirical correlation between answer accuracy and pointwise mutual information. Additionally, we propose two methods that use the pointwise mutual information between a document and a question as a gauge for selecting and constructing prompts that lead to better performance, whose effectiveness we demonstrate through experimentation.