Cross-Encoder Rediscovers a Semantic Variant of BM25
This work provides interpretability insights for neural ranking models, addressing transparency and safety concerns in information retrieval.
The paper investigates a Cross-Encoder variant of MiniLM to understand its relevance features, finding it uses a semantic variant of BM25 with localized components like Transformer attention heads for term frequency and a low-rank embedding matrix for inverse document frequency, leading to improved retrieval performance.
Neural Ranking Models (NRMs) have rapidly advanced state-of-the-art performance on information retrieval tasks. In this work, we investigate a Cross-Encoder variant of MiniLM to determine which relevance features it computes and where they are stored. We find that it employs a semantic variant of the traditional BM25 in an interpretable manner, featuring localized components: (1) Transformer attention heads that compute soft term frequency while controlling for term saturation and document length effects, and (2) a low-rank component of its embedding matrix that encodes inverse document frequency information for the vocabulary. This suggests that the Cross-Encoder uses the same fundamental mechanisms as BM25, but further leverages their capacity to capture semantics for improved retrieval performance. The granular understanding lays the groundwork for model editing to enhance model transparency, addressing safety concerns, and improving scalability in training and real-world applications.