A Medical Semantic-Assisted Transformer for Radiographic Report Generation
This work addresses the cross-domain task of automated radiographic report generation for medical imaging, which is incremental as it builds on existing methods by introducing new components to improve performance.
The paper tackled the problem of generating accurate and semantic-coherent radiographic reports from medical images by addressing challenges in capturing fine-grained visual differences and providing effective medical semantic features, resulting in a method that outperforms state-of-the-art approaches on the MIMIC-CXR benchmark.
Automated radiographic report generation is a challenging cross-domain task that aims to automatically generate accurate and semantic-coherence reports to describe medical images. Despite the recent progress in this field, there are still many challenges at least in the following aspects. First, radiographic images are very similar to each other, and thus it is difficult to capture the fine-grained visual differences using CNN as the visual feature extractor like many existing methods. Further, semantic information has been widely applied to boost the performance of generation tasks (e.g. image captioning), but existing methods often fail to provide effective medical semantic features. Toward solving those problems, in this paper, we propose a memory-augmented sparse attention block utilizing bilinear pooling to capture the higher-order interactions between the input fine-grained image features while producing sparse attention. Moreover, we introduce a novel Medical Concepts Generation Network (MCGN) to predict fine-grained semantic concepts and incorporate them into the report generation process as guidance. Our proposed method shows promising performance on the recently released largest benchmark MIMIC-CXR. It outperforms multiple state-of-the-art methods in image captioning and medical report generation.