Combining 3D Image and Tabular Data via the Dynamic Affine Feature Map Transform
This addresses the challenge of integrating low-dimensional clinical data with high-dimensional images for improved medical diagnosis, representing an incremental advancement in multimodal deep learning for healthcare.
The paper tackles the problem of combining 3D medical images and tabular data for Alzheimer's disease diagnosis and prediction, introducing the Dynamic Affine Feature Map Transform (DAFT) module that achieves a mean balanced accuracy of 0.622 and mean c-index of 0.748, outperforming competing methods.
Prior work on diagnosing Alzheimer's disease from magnetic resonance images of the brain established that convolutional neural networks (CNNs) can leverage the high-dimensional image information for classifying patients. However, little research focused on how these models can utilize the usually low-dimensional tabular information, such as patient demographics or laboratory measurements. We introduce the Dynamic Affine Feature Map Transform (DAFT), a general-purpose module for CNNs that dynamically rescales and shifts the feature maps of a convolutional layer, conditional on a patient's tabular clinical information. We show that DAFT is highly effective in combining 3D image and tabular information for diagnosis and time-to-dementia prediction, where it outperforms competing CNNs with a mean balanced accuracy of 0.622 and mean c-index of 0.748, respectively. Our extensive ablation study provides valuable insights into the architectural properties of DAFT. Our implementation is available at https://github.com/ai-med/DAFT.