Lesion Guided Explainable Few Weak-shot Medical Report Generation
This work addresses the challenge of adapting automated medical report generation to unseen diseases, which is incremental as it builds on existing methods by incorporating lesion detection and multi-view feature alignment.
The paper tackles the problem of generating medical reports for novel diseases not seen during training by proposing a lesion-guided framework that aligns visual and semantic features, and it demonstrates superior performance on the FFA-IR dataset for novel disease report generation.
Medical images are widely used in clinical practice for diagnosis. Automatically generating interpretable medical reports can reduce radiologists' burden and facilitate timely care. However, most existing approaches to automatic report generation require sufficient labeled data for training. In addition, the learned model can only generate reports for the training classes, lacking the ability to adapt to previously unseen novel diseases. To this end, we propose a lesion guided explainable few weak-shot medical report generation framework that learns correlation between seen and novel classes through visual and semantic feature alignment, aiming to generate medical reports for diseases not observed in training. It integrates a lesion-centric feature extractor and a Transformer-based report generation module. Concretely, the lesion-centric feature extractor detects the abnormal regions and learns correlations between seen and novel classes with multi-view (visual and lexical) embeddings. Then, features of the detected regions and corresponding embeddings are concatenated as multi-view input to the report generation module for explainable report generation, including text descriptions and corresponding abnormal regions detected in the images. We conduct experiments on FFA-IR, a dataset providing explainable annotations, showing that our framework outperforms others on report generation for novel diseases.