RankSHAP: Shapley Value Based Feature Attributions for Learning to Rank
This provides a consistent and axiomatically backed explanation method for practitioners in information retrieval, though it is incremental as it extends existing Shapley value approaches to ranking.
The paper tackled the problem of inconsistent feature attributions in learning to rank by proposing RankSHAP, a Shapley value-based method that satisfies fundamental axioms, and experiments on two datasets and a user study showed it aligns with human intuition.
Numerous works propose post-hoc, model-agnostic explanations for learning to rank, focusing on ordering entities by their relevance to a query through feature attribution methods. However, these attributions often weakly correlate or contradict each other, confusing end users. We adopt an axiomatic game-theoretic approach, popular in the feature attribution community, to identify a set of fundamental axioms that every ranking-based feature attribution method should satisfy. We then introduce Rank-SHAP, extending classical Shapley values to ranking. We evaluate the RankSHAP framework through extensive experiments on two datasets, multiple ranking methods and evaluation metrics. Additionally, a user study confirms RankSHAP's alignment with human intuition. We also perform an axiomatic analysis of existing rank attribution algorithms to determine their compliance with our proposed axioms. Ultimately, our aim is to equip practitioners with a set of axiomatically backed feature attribution methods for studying IR ranking models, that ensure generality as well as consistency.