Prototype-driven fusion of pathology and spatial transcriptomics for interpretable survival prediction
This work addresses the challenge of interpretable survival prediction for cancer patients by fusing pathology and spatial transcriptomics data, offering a scalable and biologically grounded approach for multimodal learning in oncology.
The paper tackled the problem of predicting survival in triple-negative breast cancer by integrating whole slide images and spatial transcriptomics, achieving strong and consistent performance across five survival endpoints with superior or comparable results to existing methods. It introduced PathoSpatial, an interpretable framework that learns spatially informed prognostic representations while enabling post-hoc prototype interpretation and molecular risk decomposition.
Whole slide images (WSIs) enable weakly supervised prognostic modeling via multiple instance learning (MIL). Spatial transcriptomics (ST) preserves in situ gene expression, providing a spatial molecular context that complements morphology. As paired WSI-ST cohorts scale to population level, leveraging their complementary spatial signals for prognosis becomes crucial; however, principled cross-modal fusion strategies remain limited for this paradigm. To this end, we introduce PathoSpatial, an interpretable end-to-end framework integrating co-registered WSIs and ST to learn spatially informed prognostic representations. PathoSpatial uses task-guided prototype learning within a multi-level experts architecture, adaptively orchestrating unsupervised within-modality discovery with supervised cross-modal aggregation. By design, PathoSpatial substantially strengthens interpretability while maintaining discriminative ability. We evaluate PathoSpatial on a triple-negative breast cancer cohort with paired ST and WSIs. PathoSpatial delivers strong and consistent performance across five survival endpoints, achieving superior or comparable performance to leading unimodal and multimodal methods. PathoSpatial inherently enables post-hoc prototype interpretation and molecular risk decomposition, providing quantitative, biologically grounded explanations, highlighting candidate prognostic factors. We present PathoSpatial as a proof-of-concept for scalable and interpretable multimodal learning for spatial omics-pathology fusion.