An XAI-based Analysis of Shortcut Learning in Neural Networks
This work addresses the issue of shortcut learning in neural networks, which is crucial for making AI models safer in practice, though it is incremental as it builds on existing XAI and spurious correlation research.
The paper tackles the problem of neural networks learning spurious features by introducing a diagnostic measure called the neuron spurious score to analyze how and where these features are encoded in CNNs and ViTs, finding that spurious features are partially disentangled with varying degrees across architectures and that existing mitigation methods have incomplete assumptions.
Machine learning models tend to learn spurious features - features that strongly correlate with target labels but are not causal. Existing approaches to mitigate models' dependence on spurious features work in some cases, but fail in others. In this paper, we systematically analyze how and where neural networks encode spurious correlations. We introduce the neuron spurious score, an XAI-based diagnostic measure to quantify a neuron's dependence on spurious features. We analyze both convolutional neural networks (CNNs) and vision transformers (ViTs) using architecture-specific methods. Our results show that spurious features are partially disentangled, but the degree of disentanglement varies across model architectures. Furthermore, we find that the assumptions behind existing mitigation methods are incomplete. Our results lay the groundwork for the development of novel methods to mitigate spurious correlations and make AI models safer to use in practice.