Mutual Attention-based Hybrid Dimensional Network for Multimodal Imaging Computer-aided Diagnosis
This work addresses the problem of improving diagnostic accuracy in multimodal 3D medical imaging for computer-aided diagnosis, though it appears incremental as it builds on existing CNN and attention methods.
The paper tackles the challenge of training competitive 3D CNNs for multimodal medical image classification with limited data by proposing MMNet, a mutual attention-based hybrid dimensional network that integrates 2D and 3D CNNs with attention mechanisms, achieving results competitive with state-of-the-art on three datasets including Parotid Gland Tumor, MRNet, and PROSTATEx.
Recent works on Multimodal 3D Computer-aided diagnosis have demonstrated that obtaining a competitive automatic diagnosis model when a 3D convolution neural network (CNN) brings more parameters and medical images are scarce remains nontrivial and challenging. Considering both consistencies of regions of interest in multimodal images and diagnostic accuracy, we propose a novel mutual attention-based hybrid dimensional network for MultiModal 3D medical image classification (MMNet). The hybrid dimensional network integrates 2D CNN with 3D convolution modules to generate deeper and more informative feature maps, and reduce the training complexity of 3D fusion. Besides, the pre-trained model of ImageNet can be used in 2D CNN, which improves the performance of the model. The stereoscopic attention is focused on building rich contextual interdependencies of the region in 3D medical images. To improve the regional correlation of pathological tissues in multimodal medical images, we further design a mutual attention framework in the network to build the region-wise consistency in similar stereoscopic regions of different image modalities, providing an implicit manner to instruct the network to focus on pathological tissues. MMNet outperforms many previous solutions and achieves results competitive to the state-of-the-art on three multimodal imaging datasets, i.e., Parotid Gland Tumor (PGT) dataset, the MRNet dataset, and the PROSTATEx dataset, and its advantages are validated by extensive experiments.