Training a Utility-based Retriever Through Shared Context Attribution for Retrieval-Augmented Language Models
This work addresses the challenge of accurately capturing passage utility for downstream tasks in retrieval-augmented language models, offering an incremental improvement over existing methods.
The paper tackles the problem of training utility-based retrievers for retrieval-augmented language models by proposing SCARLet, a framework that uses shared context and perturbation-based attribution to improve task generalization and passage utility estimation, resulting in consistent performance gains across ten datasets.
Retrieval-Augmented Language Models boost task performance, owing to the retriever that provides external knowledge. Although crucial, the retriever primarily focuses on semantics relevance, which may not always be effective for generation. Thus, utility-based retrieval has emerged as a promising topic, prioritizing passages that provides valid benefits for downstream tasks. However, due to insufficient understanding, capturing passage utility accurately remains unexplored. This work proposes SCARLet, a framework for training utility-based retrievers in RALMs, which incorporates two key factors, multi-task generalization and inter-passage interaction. First, SCARLet constructs shared context on which training data for various tasks is synthesized. This mitigates semantic bias from context differences, allowing retrievers to focus on learning task-specific utility for better task generalization. Next, SCARLet uses a perturbation-based attribution method to estimate passage-level utility for shared context, which reflects interactions between passages and provides more accurate feedback. We evaluate our approach on ten datasets across various tasks, both in-domain and out-of-domain, showing that retrievers trained by SCARLet consistently improve the overall performance of RALMs.