Probing Ranking LLMs: A Mechanistic Analysis for Information Retrieval
This work addresses the interpretability and reliability of ranking models in information retrieval, offering insights for developing more transparent systems, though it is incremental as it analyzes existing models without proposing new methods.
The study investigated the internal mechanisms of fine-tuned LLMs for passage reranking by probing neuron activations to identify encoded human-engineered and semantic features, revealing which statistical IR features are present or missing and analyzing generalization behaviors on out-of-distribution data.
Transformer networks, particularly those achieving performance comparable to GPT models, are well known for their robust feature extraction abilities. However, the nature of these extracted features and their alignment with human-engineered ones remain unexplored. In this work, we investigate the internal mechanisms of state-of-the-art, fine-tuned LLMs for passage reranking. We employ a probing-based analysis to examine neuron activations in ranking LLMs, identifying the presence of known human-engineered and semantic features. Our study spans a broad range of feature categories, including lexical signals, document structure, query-document interactions, and complex semantic representations, to uncover underlying patterns influencing ranking decisions. Through experiments on four different ranking LLMs, we identify statistical IR features that are prominently encoded in LLM activations, as well as others that are notably missing. Furthermore, we analyze how these models respond to out-of-distribution queries and documents, revealing distinct generalization behaviors. By dissecting the latent representations within LLM activations, we aim to improve both the interpretability and effectiveness of ranking models. Our findings offer crucial insights for developing more transparent and reliable retrieval systems, and we release all necessary scripts and code to support further exploration.