Generalizability vs. Counterfactual Explainability Trade-Off
This work addresses the trade-off between generalization and explainability for machine learning practitioners, offering a theoretical and empirical framework.
The paper investigates the relationship between model generalization and counterfactual explainability, showing that poor generalization increases the ease of generating counterfactual explanations, with empirical validation using ε-VCP as a proxy for overfitting.
In this work, we investigate the relationship between model generalization and counterfactual explainability in supervised learning. We introduce the notion of $\varepsilon$-valid counterfactual probability ($\varepsilon$-VCP) -- the probability of finding perturbations of a data point within its $\varepsilon$-neighborhood that result in a label change. We provide a theoretical analysis of $\varepsilon$-VCP in relation to the geometry of the model's decision boundary, showing that $\varepsilon$-VCP tends to increase with model overfitting. Our findings establish a rigorous connection between poor generalization and the ease of counterfactual generation, revealing an inherent trade-off between generalization and counterfactual explainability. Empirical results validate our theory, suggesting $\varepsilon$-VCP as a practical proxy for quantitatively characterizing overfitting.