Memory-based Cross-modal Semantic Alignment Network for Radiology Report Generation
This work addresses the problem of reducing radiologist workload and aiding disease diagnosis through automated report generation, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the challenge of generating accurate radiology reports from images by addressing the difficulty in learning latent relations due to sparse disease-related information, proposing a memory-based cross-modal semantic alignment model that achieves state-of-the-art performance on the MIMIC-CXR dataset.
Generating radiology reports automatically reduces the workload of radiologists and helps the diagnoses of specific diseases. Many existing methods take this task as modality transfer process. However, since the key information related to disease accounts for a small proportion in both image and report, it is hard for the model to learn the latent relation between the radiology image and its report, thus failing to generate fluent and accurate radiology reports. To tackle this problem, we propose a memory-based cross-modal semantic alignment model (MCSAM) following an encoder-decoder paradigm. MCSAM includes a well initialized long-term clinical memory bank to learn disease-related representations as well as prior knowledge for different modalities to retrieve and use the retrieved memory to perform feature consolidation. To ensure the semantic consistency of the retrieved cross modal prior knowledge, a cross-modal semantic alignment module (SAM) is proposed. SAM is also able to generate semantic visual feature embeddings which can be added to the decoder and benefits report generation. More importantly, to memorize the state and additional information while generating reports with the decoder, we use learnable memory tokens which can be seen as prompts. Extensive experiments demonstrate the promising performance of our proposed method which generates state-of-the-art performance on the MIMIC-CXR dataset.