The Representational Limit of Scalar Interactions: An Interventional Decomposition
For interpretability researchers, it addresses the fundamental conflation in scalar interaction measures with a principled decomposition, though the method is domain-specific.
The paper proves that scalar pairwise interaction scores conflate uniqueness, redundancy, and synergy, and introduces Stochastic Hi-Fi, a retraining-free method that decomposes per-feature predictability into these components. It recovers up to 411x larger interaction magnitudes on tabular SCMs and improves Deletion AUC on NIH ChestX-ray14.
Signed pairwise interaction scores fundamentally conflate uniqueness (U), redundancy (R), and synergy (S). We prove this on a minimal 3-way XOR structural causal model: faithful indices such as Shapley-Taylor return zero per pair, whereas projective indices such as Shapley Interaction spread the third-order effect into pair scalars that conflate the three mechanisms. We introduce Stochastic Hi-Fi, a post-hoc, retraining-free predictability decomposition that estimates per-feature U/R/S profiles by interventional masked inference. The estimator provides exact interventional semantics, finite-sample Monte Carlo bounds, strict variance reduction from coupled diamond sampling, and uniform finite-vocabulary convergence. Across tabular SCMs, Stochastic Hi-Fi recovers structure missed by scalar baselines (up to 411x larger interaction-magnitude recovery ratios). It also separates redundant and synergistic heads in the GPT-2 IOI circuit. On NIH ChestX-ray14, Stochastic Hi-Fi matches GradCAM on Pointing Game and improves substantially on Deletion AUC.