Hypothesis Class Determines Explanation: Why Accurate Models Disagree on Feature Attribution
This reveals that model selection is not explanation-neutral, impacting practices like auditing and regulatory evaluation in AI, though it is incremental in challenging a specific assumption.
The paper tackles the assumption that prediction-equivalent models produce equivalent explanations in explainable AI, showing through an empirical study across 24 datasets that models with identical predictive behavior can yield substantially different feature attributions, with cross-class pairs often showing agreement near or below the lottery threshold.
The assumption that prediction-equivalent models produce equivalent explanations underlies many practices in explainable AI, including model selection, auditing, and regulatory evaluation. In this work, we show that this assumption does not hold. Through a large-scale empirical study across 24 datasets and multiple model classes, we find that models with identical predictive behavior can produce substantially different feature attributions. This disagreement is highly structured: models within the same hypothesis class exhibit strong agreement, while cross-class pairs (e.g., tree-based vs. linear) trained on identical data splits show substantially reduced agreement, consistently near or below the lottery threshold. We identify hypothesis class as the structural driver of this phenomenon, which we term the Explanation Lottery. We theoretically show that the resulting Agreement Gap persists under interaction structure in the data-generating process. This structural finding motivates a post-hoc diagnostic, the Explanation Reliability Score R(x), which predicts when explanations are stable across architectures without additional training. Our results demonstrate that model selection is not explanation-neutral: the hypothesis class chosen for deployment can determine which features are attributed responsibility for a decision.