How GNNs Facilitate CNNs in Mining Geometric Information from Large-Scale Medical Images
This work addresses the need for more comprehensive representations in medical image analysis for oncology tasks by integrating image-level and cell-level features, though it appears incremental as it combines existing methods (CNNs and GNNs) in a novel way.
The authors tackled the problem of extracting both morphological textures and spatial geometric information from gigapixel medical images by proposing a fusion framework that combines CNNs for global image-level features with GNNs for cell-level spatial relations. Their approach achieved consistent AUC improvements of over 5% on biomarker prediction tasks for colorectal and gastric cancers compared to using CNNs or GNNs alone.
Gigapixel medical images provide massive data, both morphological textures and spatial information, to be mined. Due to the large data scale in histology, deep learning methods play an increasingly significant role as feature extractors. Existing solutions heavily rely on convolutional neural networks (CNNs) for global pixel-level analysis, leaving the underlying local geometric structure such as the interaction between cells in the tumor microenvironment unexplored. The topological structure in medical images, as proven to be closely related to tumor evolution, can be well characterized by graphs. To obtain a more comprehensive representation for downstream oncology tasks, we propose a fusion framework for enhancing the global image-level representation captured by CNNs with the geometry of cell-level spatial information learned by graph neural networks (GNN). The fusion layer optimizes an integration between collaborative features of global images and cell graphs. Two fusion strategies have been developed: one with MLP which is simple but turns out efficient through fine-tuning, and the other with Transformer gains a champion in fusing multiple networks. We evaluate our fusion strategies on histology datasets curated from large patient cohorts of colorectal and gastric cancers for three biomarker prediction tasks. Both two models outperform plain CNNs or GNNs, reaching a consistent AUC improvement of more than 5% on various network backbones. The experimental results yield the necessity for combining image-level morphological features with cell spatial relations in medical image analysis. Codes are available at https://github.com/yiqings/HEGnnEnhanceCnn.