Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts
This addresses a domain-specific challenge in medical imaging for improved diagnostic or surgical planning, with incremental advancements in multimodal fusion.
The paper tackles the problem of synthesizing missing medical images from observed images across different modalities, achieving high-resolution synthesis for brain MRI and ultrasound imaging.
We propose a deep mixture of multimodal hierarchical variational auto-encoders called MMHVAE that synthesizes missing images from observed images in different modalities. MMHVAE's design focuses on tackling four challenges: (i) creating a complex latent representation of multimodal data to generate high-resolution images; (ii) encouraging the variational distributions to estimate the missing information needed for cross-modal image synthesis; (iii) learning to fuse multimodal information in the context of missing data; (iv) leveraging dataset-level information to handle incomplete data sets at training time. Extensive experiments are performed on the challenging problem of pre-operative brain multi-parametric magnetic resonance and intra-operative ultrasound imaging.