LIMITR: Leveraging Local Information for Medical Image-Text Representation
This work addresses the challenge of medical image-text analysis for healthcare applications, but it appears incremental as it builds on existing representation learning methods with domain-specific adaptations.
The paper tackles the problem of learning joint representations for chest X-ray images and radiological reports by introducing a model with a novel alignment scheme that incorporates local and global information, along with domain-specific features, resulting in improved performance on retrieval tasks such as text-image retrieval, class-based retrieval, and phrase-grounding.
Medical imaging analysis plays a critical role in the diagnosis and treatment of various medical conditions. This paper focuses on chest X-ray images and their corresponding radiological reports. It presents a new model that learns a joint X-ray image & report representation. The model is based on a novel alignment scheme between the visual data and the text, which takes into account both local and global information. Furthermore, the model integrates domain-specific information of two types -- lateral images and the consistent visual structure of chest images. Our representation is shown to benefit three types of retrieval tasks: text-image retrieval, class-based retrieval, and phrase-grounding.