TMSS: An End-to-End Transformer-based Multimodal Network for Segmentation and Survival Prediction
This work addresses survival prediction for cancer patients by integrating multimodal data in a novel way, though it is incremental as it builds on existing transformer and multimodal methods.
The authors tackled the problem of cancer patient survival prediction by developing an end-to-end transformer-based multimodal network that mimics oncologists' fusion of medical images and patient history, achieving a concordance index of 0.763±0.14 for prognosis and a dice score of 0.772±0.030 for segmentation.
When oncologists estimate cancer patient survival, they rely on multimodal data. Even though some multimodal deep learning methods have been proposed in the literature, the majority rely on having two or more independent networks that share knowledge at a later stage in the overall model. On the other hand, oncologists do not do this in their analysis but rather fuse the information in their brain from multiple sources such as medical images and patient history. This work proposes a deep learning method that mimics oncologists' analytical behavior when quantifying cancer and estimating patient survival. We propose TMSS, an end-to-end Transformer based Multimodal network for Segmentation and Survival prediction that leverages the superiority of transformers that lies in their abilities to handle different modalities. The model was trained and validated for segmentation and prognosis tasks on the training dataset from the HEad & NeCK TumOR segmentation and the outcome prediction in PET/CT images challenge (HECKTOR). We show that the proposed prognostic model significantly outperforms state-of-the-art methods with a concordance index of 0.763+/-0.14 while achieving a comparable dice score of 0.772+/-0.030 to a standalone segmentation model. The code is publicly available.