C^2M-DoT: Cross-modal consistent multi-view medical report generation with domain transfer network
This addresses the challenge of multi-view medical report generation in clinical practice, where existing methods are limited by reliance on multi-view data during inference, offering an incremental improvement with domain transfer for single-view scenarios.
The paper tackled the problem of generating medical reports from multiple medical images by proposing C^2M-DoT, which uses cross-modal consistency and domain transfer to handle single-view inputs, achieving substantial performance improvements over state-of-the-art baselines on two public datasets.
In clinical scenarios, multiple medical images with different views are usually generated simultaneously, and these images have high semantic consistency. However, most existing medical report generation methods only consider single-view data. The rich multi-view mutual information of medical images can help generate more accurate reports, however, the dependence of multi-view models on multi-view data in the inference stage severely limits their application in clinical practice. In addition, word-level optimization based on numbers ignores the semantics of reports and medical images, and the generated reports often cannot achieve good performance. Therefore, we propose a cross-modal consistent multi-view medical report generation with a domain transfer network (C^2M-DoT). Specifically, (i) a semantic-based multi-view contrastive learning medical report generation framework is adopted to utilize cross-view information to learn the semantic representation of lesions; (ii) a domain transfer network is further proposed to ensure that the multi-view report generation model can still achieve good inference performance under single-view input; (iii) meanwhile, optimization using a cross-modal consistency loss facilitates the generation of textual reports that are semantically consistent with medical images. Extensive experimental studies on two public benchmark datasets demonstrate that C^2M-DoT substantially outperforms state-of-the-art baselines in all metrics. Ablation studies also confirmed the validity and necessity of each component in C^2M-DoT.