OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries
For researchers in information retrieval and NLP, this work highlights a previously overlooked class of queries where retrieval, not verification, is the bottleneck, motivating new architectures.
The paper identifies 'oblique' queries that require retrieving documents matching latent patterns (e.g., implicit stances, failure modes), and introduces OBLIQ-Bench, a suite of five such tasks. It shows that while reasoning LLMs can verify relevance, current retrieval pipelines fail to surface most relevant documents, exposing a key bottleneck.
Retrieval benchmarks are increasingly saturating, but we argue that efficient search is far from a solved problem. We identify a class of queries we call oblique, which seek documents that instantiate a latent pattern, like finding all tweets that express an implicit stance, chat logs that demonstrate a particular failure mode, or transcripts that match an abstract scenario. We study three mechanisms through which obliqueness may arise and introduce OBLIQ-Bench, a suite of five oblique search problems over real long-tail corpora. OBLIQ-Bench exposes an overlooked asymmetry between retrieval and verification, where reasoning LLMs reliably recognize latent relevance whenever relevant documents are surfaced, but even sophisticated retrieval pipelines fail to surface most relevant documents in the first place. We hope that OBLIQ-Bench will drive research into retrieval architectures that efficiently capture latent patterns and implicit signals in large corpora.