Entity Labels Are Not Entity Signals: A Framework for Observable Relevance in Document Re-Ranking
For researchers in entity-aware document retrieval, this paper identifies a fundamental flaw in current assumptions and provides a practical framework to improve retrieval effectiveness.
Entity-aware retrieval assumes semantically relevant entities are useful ranking signals, but this paper shows that entity links are noisy hypotheses, not ground-truth observations. They formalize Conceptual vs. Observable Entity Relevance, finding near-chance agreement between them, and show that aligning supervision with observable relevance improves non-relevant pruning by up to 10x and MAP by 0.051 over BM25.
Entity-aware document retrieval uses query-associated entities as ranking signals, assuming that semantically relevant entities are also useful retrieval signals. We show this assumption is insufficient- and explain why. Unlike terms, which are ground-truth observations, entity links are hypotheses produced by an imperfect linker: an entity can be topically central yet provide no discriminative signal if the linker fires indiscriminately across relevant and non-relevant documents. We formalize this as a distinction between Conceptual Entity Relevance (CER)- whether an entity is topically related to a query- and Observable Entity Relevance (OER)- whether its observed presence in a collection discriminates relevant from non-relevant documents. Across four collections and annotation sources including human entity judgments, CER and OER exhibit near-chance agreement ($κ\approx 0$), while OER operationalizations agree substantially ($κ\approx 0.5$), confirming CER as the systematic outlier. CER-based supervision selects topically plausible but weakly discriminative entities, pruning fewer than 4% of non-relevant documents on some collections. Aligning supervision with OER improves non-relevant pruning by up to 10x and open-world MAP by 0.051 over BM25. Our findings motivate a shift from conceptual to observable notions of entity relevance in entity-aware retrieval.