Segmentation Ability Map: Interpret deep features for medical image segmentation
This work addresses the need for interpretability in deep learning for medical image segmentation, providing tools to understand hidden representations and estimate performance without ground-truth, though it is incremental in applying existing interpretability concepts to segmentation.
The paper tackles the problem of interpreting deep features in medical image segmentation by proposing a prototype segmentation method to compute binary segmentation maps and quantify feature segmentation abilities with Dice scores, applied across multiple medical imaging tasks including brain MRI, skin lesions, COVID-19 CT, prostate MRI, and pancreatic CT.
Deep convolutional neural networks (CNNs) have been widely used for medical image segmentation. In most studies, only the output layer is exploited to compute the final segmentation results and the hidden representations of the deep learned features have not been well understood. In this paper, we propose a prototype segmentation (ProtoSeg) method to compute a binary segmentation map based on deep features. We measure the segmentation abilities of the features by computing the Dice between the feature segmentation map and ground-truth, named as the segmentation ability score (SA score for short). The corresponding SA score can quantify the segmentation abilities of deep features in different layers and units to understand the deep neural networks for segmentation. In addition, our method can provide a mean SA score which can give a performance estimation of the output on the test images without ground-truth. Finally, we use the proposed ProtoSeg method to compute the segmentation map directly on input images to further understand the segmentation ability of each input image. Results are presented on segmenting tumors in brain MRI, lesions in skin images, COVID-related abnormality in CT images, prostate segmentation in abdominal MRI, and pancreatic mass segmentation in CT images. Our method can provide new insights for interpreting and explainable AI systems for medical image segmentation. Our code is available on: \url{https://github.com/shengfly/ProtoSeg}.