Who Does What in Deep Learning? Multidimensional Game-Theoretic Attribution of Function of Neural Units
This addresses the need for interpretability in large neural networks for researchers and practitioners, though it is an incremental extension of game-theoretic methods to a new context.
The paper tackles the problem of attributing contributions of individual neural units to high-dimensional outputs in deep learning models, introducing Multiperturbation Shapley-value Analysis (MSA) to generate unit-wise contribution maps that match output dimensionality, applied to models up to 56 billion parameters, revealing insights like computation concentration and language-specific experts.
Neural networks now generate text, images, and speech with billions of parameters, producing a need to know how each neural unit contributes to these high-dimensional outputs. Existing explainable-AI methods, such as SHAP, attribute importance to inputs, but cannot quantify the contributions of neural units across thousands of output pixels, tokens, or logits. Here we close that gap with Multiperturbation Shapley-value Analysis (MSA), a model-agnostic game-theoretic framework. By systematically lesioning combinations of units, MSA yields Shapley Modes, unit-wise contribution maps that share the exact dimensionality of the model's output. We apply MSA across scales, from multi-layer perceptrons to the 56-billion-parameter Mixtral-8x7B and Generative Adversarial Networks (GAN). The approach demonstrates how regularisation concentrates computation in a few hubs, exposes language-specific experts inside the LLM, and reveals an inverted pixel-generation hierarchy in GANs. Together, these results showcase MSA as a powerful approach for interpreting, editing, and compressing deep neural networks.