Medical Image Captioning via Generative Pretrained Transformers
This addresses the problem of generating comprehensive radiology records for medical professionals, though it appears incremental as it combines existing models.
The paper tackles automatic clinical caption generation for chest X-Ray scans by combining Show-Attend-Tell and GPT-3 models to generate textual summaries with pathology information and localization heatmaps, achieving results validated on medical datasets like Open-I and MIMIC-CXR with natural language assessment metrics.
The automatic clinical caption generation problem is referred to as proposed model combining the analysis of frontal chest X-Ray scans with structured patient information from the radiology records. We combine two language models, the Show-Attend-Tell and the GPT-3, to generate comprehensive and descriptive radiology records. The proposed combination of these models generates a textual summary with the essential information about pathologies found, their location, and the 2D heatmaps localizing each pathology on the original X-Ray scans. The proposed model is tested on two medical datasets, the Open-I, MIMIC-CXR, and the general-purpose MS-COCO. The results measured with the natural language assessment metrics prove their efficient applicability to the chest X-Ray image captioning.