Multimodal self-supervised learning for lesion localization
This work addresses the problem of lesion localization in medical imaging for clinicians, offering an incremental improvement by enhancing fine-grained semantic alignment between images and reports.
The paper tackled the challenge of accurately localizing lesions in medical images without detailed positional annotations by introducing a multimodal self-supervised learning method that uses full sentences from textual reports for local semantic alignment. The method achieved leading results on multiple datasets, confirming its efficacy in lesion localization.
Multimodal deep learning utilizing imaging and diagnostic reports has made impressive progress in the field of medical imaging diagnostics, demonstrating a particularly strong capability for auxiliary diagnosis in cases where sufficient annotation information is lacking. Nonetheless, localizing diseases accurately without detailed positional annotations remains a challenge. Although existing methods have attempted to utilize local information to achieve fine-grained semantic alignment, their capability in extracting the fine-grained semantics of the comprehensive context within reports is limited. To address this problem, a new method is introduced that takes full sentences from textual reports as the basic units for local semantic alignment. This approach combines chest X-ray images with their corresponding textual reports, performing contrastive learning at both global and local levels. The leading results obtained by this method on multiple datasets confirm its efficacy in the task of lesion localization.