Measuring Feature Dependency of Neural Networks by Collapsing Feature Dimensions in the Data Manifold
This provides a tool for researchers and practitioners to interpret neural network decisions in domains like medical imaging, though it is incremental as it builds on existing manifold and generative model techniques.
The paper tackles the problem of understanding which human-interpretable features neural networks rely on by introducing a method that removes a feature by collapsing its dimension in the data manifold and measuring performance drop. They tested it on synthetic images, Alzheimer's disease prediction with MRI data, and cell nuclei classification, showing it can identify feature dependencies in these tasks.
This paper introduces a new technique to measure the feature dependency of neural network models. The motivation is to better understand a model by querying whether it is using information from human-understandable features, e.g., anatomical shape, volume, or image texture. Our method is based on the principle that if a model is dependent on a feature, then removal of that feature should significantly harm its performance. A targeted feature is "removed" by collapsing the dimension in the data distribution that corresponds to that feature. We perform this by moving data points along the feature dimension to a baseline feature value while staying on the data manifold, as estimated by a deep generative model. Then we observe how the model's performance changes on the modified test data set, with the target feature dimension removed. We test our method on deep neural network models trained on synthetic image data with known ground truth, an Alzheimer's disease prediction task using MRI and hippocampus segmentations from the OASIS-3 dataset, and a cell nuclei classification task using the Lizard dataset.