UsefulBench: Towards Decision-Useful Information as a Target for Information Retrieval
For information retrieval researchers, this work highlights a gap between relevance and practical usefulness, but the dataset is domain-specific and the improvements are incremental.
The paper introduces UsefulBench, a dataset distinguishing relevance from usefulness in information retrieval, showing that similarity-based methods align with relevance while LLMs struggle with domain-specific usefulness, achieving partial improvements.
Conventional information retrieval is concerned with identifying the relevance of texts for a given query. Yet, the conventional definition of relevance is dominated by aspects of similarity in texts, leaving unobserved whether the text is truly useful for addressing the query. For instance, when answering whether Paris is larger than Berlin, texts about Paris being in France are relevant (lexical/semantic similarity), but not useful. In this paper, we introduce UsefulBench, a domain-specific dataset curated by three professional analysts labeling whether a text is connected to a query (relevance) or holds practical value in responding to it (usefulness). We show that classic similarity-based information retrieval aligns more strongly with relevance. While LLM-based systems can counteract this bias, we find that domain-specific problems require a high degree of expertise, which current LLMs do not fully incorporate. We explore approaches to (partially) overcome this challenge. However, UsefulBench presents a dataset challenge for targeted information retrieval systems.