MedRAT: Unpaired Medical Report Generation via Auxiliary Tasks
This addresses a challenge in medical imaging for healthcare by enabling report generation without costly paired datasets, though it is incremental as it builds on existing unpaired methods.
The paper tackles the problem of generating medical reports from X-ray images without paired training data by proposing MedRAT, a model that uses auxiliary tasks like contrastive learning and classification to align images and reports, achieving state-of-the-art results.
Medical report generation from X-ray images is a challenging task, particularly in an unpaired setting where paired image-report data is unavailable for training. To address this challenge, we propose a novel model that leverages the available information in two distinct datasets, one comprising reports and the other consisting of images. The core idea of our model revolves around the notion that combining auto-encoding report generation with multi-modal (report-image) alignment can offer a solution. However, the challenge persists regarding how to achieve this alignment when pair correspondence is absent. Our proposed solution involves the use of auxiliary tasks, particularly contrastive learning and classification, to position related images and reports in close proximity to each other. This approach differs from previous methods that rely on pre-processing steps, such as using external information stored in a knowledge graph. Our model, named MedRAT, surpasses previous state-of-the-art methods, demonstrating the feasibility of generating comprehensive medical reports without the need for paired data or external tools.