PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reporting
This work addresses the challenge of generating localized findings for PET/CT reports, which is incremental in extending vision-language models to 3D medical imaging.
The authors tackled automated reporting for 3D PET/CT scans by developing PETAR-4B, a mask-aware vision-language model that integrates PET, CT, and lesion contours, resulting in improved report generation quality as demonstrated through evaluations.
Recent advances in vision-language models (VLMs) have enabled impressive multimodal reasoning, yet most medical applications remain limited to 2D imaging. In this work, we extend VLMs to 3D positron emission tomography and computed tomography (PET/CT), a domain characterized by large volumetric data, small and dispersed lesions, and lengthy radiology reports. We introduce a large-scale dataset comprising over 11,000 lesion-level descriptions paired with 3D segmentations from more than 5,000 PET/CT exams, extracted via a hybrid rule-based and large language model (LLM) pipeline. Building upon this dataset, we propose PETAR-4B, a 3D mask-aware vision-language model that integrates PET, CT, and lesion contours for spatially grounded report generation. PETAR bridges global contextual reasoning with fine-grained lesion awareness, producing clinically coherent and localized findings. Comprehensive automated and human evaluations demonstrate that PETAR substantially improves PET/CT report generation quality, advancing 3D medical vision-language understanding.