DINOv3-Guided Cross Fusion Framework for Semantic-aware CT generation from MRI and CBCT
This work addresses the need for efficient and accurate CT synthesis in adaptive radiotherapy, offering a novel approach that could enhance treatment planning for medical professionals.
The paper tackled the problem of generating synthetic CT images from MRI or CBCT for radiation dose planning by proposing a DINOv3-Guided Cross Fusion framework, which achieved state-of-the-art performance on the SynthRAD2023 pelvic dataset with improved MS-SSIM, PSNR, and segmentation-based metrics.
Generating synthetic CT images from CBCT or MRI has a potential for efficient radiation dose planning and adaptive radiotherapy. However, existing CNN-based models lack global semantic understanding, while Transformers often overfit small medical datasets due to high model capacity and weak inductive bias. To address these limitations, we propose a DINOv3-Guided Cross Fusion (DGCF) framework that integrates a frozen self-supervised DINOv3 Transformer with a trainable CNN encoder-decoder. It hierarchically fuses global representation of Transformer and local features of CNN via a learnable cross fusion module, achieving balanced local appearance and contextual representation. Furthermore, we introduce a Multi-Level DINOv3 Perceptual (MLDP) loss that encourages semantic similarity between synthetic CT and the ground truth in DINOv3's feature space. Experiments on the SynthRAD2023 pelvic dataset demonstrate that DGCF achieved state-of-the-art performance in terms of MS-SSIM, PSNR and segmentation-based metrics on both MRI$\rightarrow$CT and CBCT$\rightarrow$CT translation tasks. To the best of our knowledge, this is the first work to employ DINOv3 representations for medical image translation, highlighting the potential of self-supervised Transformer guidance for semantic-aware CT synthesis. The code is available at https://github.com/HiLab-git/DGCF.