MedRegion-CT: Region-Focused Multimodal LLM for Comprehensive 3D CT Report Generation
This work addresses the problem of generating more accurate and clinically relevant CT reports for medical professionals, though it appears incremental as it builds on existing datasets and multimodal LLM frameworks.
The paper tackles the problem of generating comprehensive 3D CT reports by addressing the limitation of existing methods that focus on global features and miss region-specific details, resulting in MedRegion-CT achieving state-of-the-art performance on the RadGenome-Chest CT benchmark with improved natural language generation quality and clinical relevance.
The recent release of RadGenome-Chest CT has significantly advanced CT-based report generation. However, existing methods primarily focus on global features, making it challenging to capture region-specific details, which may cause certain abnormalities to go unnoticed. To address this, we propose MedRegion-CT, a region-focused Multi-Modal Large Language Model (MLLM) framework, featuring three key innovations. First, we introduce Region Representative ($R^2$) Token Pooling, which utilizes a 2D-wise pretrained vision model to efficiently extract 3D CT features. This approach generates global tokens representing overall slice features and region tokens highlighting target areas, enabling the MLLM to process comprehensive information effectively. Second, a universal segmentation model generates pseudo-masks, which are then processed by a mask encoder to extract region-centric features. This allows the MLLM to focus on clinically relevant regions, using six predefined region masks. Third, we leverage segmentation results to extract patient-specific attributions, including organ size, diameter, and locations. These are converted into text prompts, enriching the MLLM's understanding of patient-specific contexts. To ensure rigorous evaluation, we conducted benchmark experiments on report generation using the RadGenome-Chest CT. MedRegion-CT achieved state-of-the-art performance, outperforming existing methods in natural language generation quality and clinical relevance while maintaining interpretability. The code for our framework is publicly available.