Interpretable Multimodal Cancer Prototyping with Whole Slide Images and Incompletely Paired Genomics
This addresses a critical problem in precision oncology by enabling effective multimodal integration in real-world clinical scenarios with missing genomics data, representing an incremental improvement over existing methods.
The paper tackles the challenge of integrating histology and genomics for precision oncology when genomics data is incomplete, proposing a multimodal prototyping framework that achieves consistent superiority over state-of-the-art methods on multiple downstream tasks.
Multimodal approaches that integrate histology and genomics hold strong potential for precision oncology. However, phenotypic and genotypic heterogeneity limits the quality of intra-modal representations and hinders effective inter-modal integration. Furthermore, most existing methods overlook real-world clinical scenarios where genomics may be partially missing or entirely unavailable. We propose a flexible multimodal prototyping framework to integrate whole slide images and incomplete genomics for precision oncology. Our approach has four key components: 1) Biological Prototyping using text prompting and prototype-wise weighting; 2) Multiview Alignment through sample- and distribution-wise alignments; 3) Bipartite Fusion to capture both shared and modality-specific information for multimodal fusion; and 4) Semantic Genomics Imputation to handle missing data. Extensive experiments demonstrate the consistent superiority of the proposed method compared to other state-of-the-art approaches on multiple downstream tasks. The code is available at https://github.com/helenypzhang/Interpretable-Multimodal-Prototyping.