Cross-Domain Image Synthesis: Generating H&E from Multiplex Biomarker Imaging
This work addresses the integration of molecular and morphological analysis in medical imaging, offering a pathway to apply existing H&E-based diagnostic tools to rich mIF data, though it is incremental as it builds on existing generative models.
The paper tackled the problem of generating virtual H&E stains from multiplex immunofluorescence (mIF) images to provide morphological context and enable the use of H&E-based computer-aided diagnosis tools on molecular data, with results showing that a multi-level VQGAN produced images that led to superior performance in downstream nuclei segmentation and tissue classification tasks compared to a cGAN baseline.
While multiplex immunofluorescence (mIF) imaging provides deep, spatially-resolved molecular data, integrating this information with the morphological standard of Hematoxylin & Eosin (H&E) can be very important for obtaining complementary information about the underlying tissue. Generating a virtual H&E stain from mIF data offers a powerful solution, providing immediate morphological context. Crucially, this approach enables the application of the vast ecosystem of H&E-based computer-aided diagnosis (CAD) tools to analyze rich molecular data, bridging the gap between molecular and morphological analysis. In this work, we investigate the use of a multi-level Vector-Quantized Generative Adversarial Network (VQGAN) to create high-fidelity virtual H&E stains from mIF images. We rigorously evaluated our VQGAN against a standard conditional GAN (cGAN) baseline on two publicly available colorectal cancer datasets, assessing performance on both image similarity and functional utility for downstream analysis. Our results show that while both architectures produce visually plausible images, the virtual stains generated by our VQGAN provide a more effective substrate for computer-aided diagnosis. Specifically, downstream nuclei segmentation and semantic preservation in tissue classification tasks performed on VQGAN-generated images demonstrate superior performance and agreement with ground-truth analysis compared to those from the cGAN. This work establishes that a multi-level VQGAN is a robust and superior architecture for generating scientifically useful virtual stains, offering a viable pathway to integrate the rich molecular data of mIF into established and powerful H&E-based analytical workflows.