Self-Knowledge Guided Retrieval Augmentation for Large Language Models
This addresses the issue of knowledge gaps and inefficiencies in LLMs for tasks like question answering, though it is incremental as it builds on existing retrieval augmentation methods.
The paper tackles the problem of incomplete and static knowledge in large language models by proposing Self-Knowledge guided Retrieval augmentation (SKR), which adaptively uses external retrieval based on the model's self-assessment, resulting in improved performance over chain-of-thought and fully retrieval-based methods on multiple datasets with InstructGPT or ChatGPT.
Large language models (LLMs) have shown superior performance without task-specific fine-tuning. Despite the success, the knowledge stored in the parameters of LLMs could still be incomplete and difficult to update due to the computational costs. As complementary, retrieval-based methods can offer non-parametric world knowledge and improve the performance on tasks such as question answering. However, we find that the retrieved knowledge does not always help and even has a negative impact on original responses occasionally. To better make use of both internal knowledge and external world knowledge, we investigate eliciting the model's ability to recognize what they know and do not know (which is also called self-knowledge) and propose Self-Knowledge guided Retrieval augmentation (SKR), a simple yet effective method which can let LLMs refer to the questions they have previously encountered and adaptively call for external resources when dealing with new questions. We evaluate SKR on multiple datasets and demonstrate that it outperforms chain-of-thought based and fully retrieval-based methods by using either InstructGPT or ChatGPT.