A Reality Check on Context Utilisation for Retrieval-Augmented Generation
This work addresses the need for real-world aligned studies to improve RAG performance in practical applications, though it is incremental as it focuses on dataset creation and evaluation rather than a new method.
The paper tackled the problem of evaluating retrieval-augmented generation (RAG) systems by showing that synthetic datasets exaggerate context characteristics, leading to inflated performance metrics, and introduced DRUID, a real-world dataset that reveals smaller correlations between context properties and utilization scores compared to synthetic ones.
Retrieval-augmented generation (RAG) helps address the limitations of parametric knowledge embedded within a language model (LM). In real world settings, retrieved information can vary in complexity, yet most investigations of LM utilisation of context has been limited to synthetic text. We introduce DRUID (Dataset of Retrieved Unreliable, Insufficient and Difficult-to-understand contexts) with real-world queries and contexts manually annotated for stance. The dataset is based on the prototypical task of automated claim verification, for which automated retrieval of real-world evidence is crucial. We compare DRUID to synthetic datasets (CounterFact, ConflictQA) and find that artificial datasets often fail to represent the complexity and diversity of realistically retrieved context. We show that synthetic datasets exaggerate context characteristics rare in real retrieved data, which leads to inflated context utilisation results, as measured by our novel ACU score. Moreover, while previous work has mainly focused on singleton context characteristics to explain context utilisation, correlations between singleton context properties and ACU on DRUID are surprisingly small compared to other properties related to context source. Overall, our work underscores the need for real-world aligned context utilisation studies to represent and improve performance in real-world RAG settings.