Beyond Relevance: Utility-Centric Retrieval in the LLM Era
This work addresses the problem of aligning retrieval systems with the needs of LLM-based applications, offering a conceptual shift for researchers and practitioners in information retrieval and AI.
The paper argues that retrieval systems should shift from optimizing for topical relevance to utility, which measures how retrieved information helps users accomplish tasks, especially in the context of retrieval-augmented generation (RAG) where documents serve as evidence for large language models (LLMs). It presents a unified framework covering LLM-agnostic versus LLM-specific utility and provides conceptual foundations for designing retrieval systems aligned with LLM-based information access.
Information retrieval systems have traditionally optimized for topical relevance-the degree to which retrieved documents match a query. However, relevance only approximates a deeper goal: utility, namely, whether retrieved information helps accomplish a user's underlying task. The emergence of retrieval-augmented generation (RAG) fundamentally changes this paradigm. Retrieved documents are no longer consumed directly by users but instead serve as evidence for large language models (LLMs) that produce answers. As a result, retrieval effectiveness must be evaluated by its contribution to generation quality rather than by relevance-based ranking metrics alone. This tutorial argues that retrieval objectives are evolving from relevance-centric optimization toward LLM-centric utility. We present a unified framework covering LLM-agnostic versus LLM-specific utility, context-independent versus context-dependent utility, and the connection with LLM information needs and agentic RAG. By synthesizing recent advances, the tutorial provides conceptual foundations and practical guidance for designing retrieval systems aligned with the requirements of LLM-based information access.