Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignment
This work addresses the heavy workload for radiologists in clinics by automating report generation, though it appears incremental as it builds on existing multi-modal methods with specific enhancements.
The authors tackled the problem of automatically generating radiology reports from chest x-rays to reduce the burden on radiologists, and their approach outperformed state-of-the-art methods on public datasets like IU-Xray and MIMIC-CXR.
In clinics, a radiology report is crucial for guiding a patient's treatment. However, writing radiology reports is a heavy burden for radiologists. To this end, we present an automatic, multi-modal approach for report generation from a chest x-ray. Our approach, motivated by the observation that the descriptions in radiology reports are highly correlated with specific information of the x-ray images, features two distinct modules: (i) Learned knowledge base: To absorb the knowledge embedded in the radiology reports, we build a knowledge base that can automatically distil and restore medical knowledge from textual embedding without manual labour; (ii) Multi-modal alignment: to promote the semantic alignment among reports, disease labels, and images, we explicitly utilize textual embedding to guide the learning of the visual feature space. We evaluate the performance of the proposed model using metrics from both natural language generation and clinic efficacy on the public IU-Xray and MIMIC-CXR datasets. Our ablation study shows that each module contributes to improving the quality of generated reports. Furthermore, with the assistance of both modules, our approach outperforms state-of-the-art methods over almost all the metrics.