Improving Medical Visual Representations via Radiology Report Generation
This work addresses the need for better medical image analysis tools for healthcare professionals, but it is incremental as it builds on existing vision-language pretraining methods.
The paper tackled the problem of improving medical visual representations by introducing RadTex, a CNN-encoder transformer-decoder architecture optimized for radiology, and demonstrated that its captioning pretraining achieves a CheXpert macro-AUC of 89.4% and generates radiology reports with a macro-F1 score of 0.349.
Vision-language pretraining has been shown to produce high-quality visual encoders which transfer efficiently to downstream computer vision tasks. Contrastive learning approaches have increasingly been adopted for medical vision language pretraining (MVLP), yet recent developments in generative AI offer new modeling alternatives. This paper introduces RadTex, a CNN-encoder transformer-decoder architecture optimized for radiology. We explore bidirectional captioning as an alternative MVLP strategy and demonstrate that RadTex's captioning pretraining is competitive with established contrastive methods, achieving a CheXpert macro-AUC of 89.4%. Additionally, RadTex's lightweight text decoder not only generates clinically relevant radiology reports (macro-F1 score of 0.349), but also provides targeted, interactive responses, highlighting the utility of bidirectional captioning in advancing medical image analysis.