Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations
This work addresses the challenge of incomplete multi-modal image sets in medical imaging, offering a solution for scenarios like intra-operative guidance, though it appears incremental as it builds on existing multi-modal VAE approaches.
The paper tackles the problem of synthesizing missing images from multiple modalities, specifically for joint intra-operative ultrasound and Magnetic Resonance synthesis, by introducing MHVAE, a hierarchical VAE that outperformed existing methods like multi-modal VAEs, conditional GANs, and ResViT.
We introduce MHVAE, a deep hierarchical variational auto-encoder (VAE) that synthesizes missing images from various modalities. Extending multi-modal VAEs with a hierarchical latent structure, we introduce a probabilistic formulation for fusing multi-modal images in a common latent representation while having the flexibility to handle incomplete image sets as input. Moreover, adversarial learning is employed to generate sharper images. Extensive experiments are performed on the challenging problem of joint intra-operative ultrasound (iUS) and Magnetic Resonance (MR) synthesis. Our model outperformed multi-modal VAEs, conditional GANs, and the current state-of-the-art unified method (ResViT) for synthesizing missing images, demonstrating the advantage of using a hierarchical latent representation and a principled probabilistic fusion operation. Our code is publicly available \url{https://github.com/ReubenDo/MHVAE}.