Retrieval Augmented Question Answering: When Should LLMs Admit Ignorance?
This addresses a bottleneck in retrieval-augmented generation for open-domain QA, but it is incremental as it builds on existing methods with a novel prompting adjustment.
The paper tackles the problem of irrelevant information degrading performance in retrieval-augmented question answering with LLMs by proposing an adaptive prompting strategy that splits retrieved data into chunks, which matches standard prompting performance while using fewer tokens on three datasets.
The success of expanded context windows in Large Language Models (LLMs) has driven increased use of broader context in retrieval-augmented generation. We investigate the use of LLMs for retrieval augmented question answering. While longer contexts make it easier to incorporate targeted knowledge, they introduce more irrelevant information that hinders the model's generation process and degrades its performance. To address the issue, we design an adaptive prompting strategy which involves splitting the retrieved information into smaller chunks and sequentially prompting a LLM to answer the question using each chunk. Adjusting the chunk size allows a trade-off between incorporating relevant information and reducing irrelevant information. Experimental results on three open-domain question answering datasets demonstrate that the adaptive strategy matches the performance of standard prompting while using fewer tokens. Our analysis reveals that when encountering insufficient information, the LLM often generates incorrect answers instead of declining to respond, which constitutes a major source of error. This finding highlights the need for further research into enhancing LLMs' ability to effectively decline requests when faced with inadequate information.