Ablation of a Robot's Brain: Neural Networks Under a Knife
This work addresses the interpretability challenge in AI for researchers, though it appears incremental as it adapts existing neuroscience techniques to neural networks.
The paper tackled the problem of understanding how neural networks solve complex tasks by developing a novel ablation-based method to analyze information structure within networks, revealing similarities between biological and artificial neural networks.
It is still not fully understood exactly how neural networks are able to solve the complex tasks that have recently pushed AI research forward. We present a novel method for determining how information is structured inside a neural network. Using ablation (a neuroscience technique for cutting away parts of a brain to determine their function), we approach several neural network architectures from a biological perspective. Through an analysis of this method's results, we examine important similarities between biological and artificial neural networks to search for the implicit knowledge locked away in the network's weights.