QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training
This addresses the need for generalist models in clinical specialties, offering significant performance gains but is incremental in applying reinforcement learning to multimodal medical data.
The paper tackles the problem of clinical decision-making requiring reasoning across heterogeneous data by introducing QoQ-Med, a multimodal clinical foundation model that jointly reasons across medical images, time-series signals, and text reports, achieving a 43% boost in diagnostic macro-F1 and highlighting salient regions with an IoU 10x higher than open models.
Clinical decision-making routinely demands reasoning over heterogeneous data, yet existing multimodal language models (MLLMs) remain largely vision-centric and fail to generalize across clinical specialties. To bridge this gap, we introduce QoQ-Med-7B/32B, the first open generalist clinical foundation model that jointly reasons across medical images, time-series signals, and text reports. QoQ-Med is trained with Domain-aware Relative Policy Optimization (DRPO), a novel reinforcement-learning objective that hierarchically scales normalized rewards according to domain rarity and modality difficulty, mitigating performance imbalance caused by skewed clinical data distributions. Trained on 2.61 million instruction tuning pairs spanning 9 clinical domains, we show that DRPO training boosts diagnostic performance by 43% in macro-F1 on average across all visual domains as compared to other critic-free training methods like GRPO. Furthermore, with QoQ-Med trained on intensive segmentation data, it is able to highlight salient regions related to the diagnosis, with an IoU 10x higher than open models while reaching the performance of OpenAI o4-mini. To foster reproducibility and downstream research, we release (i) the full model weights, (ii) the modular training pipeline, and (iii) all intermediate reasoning traces at https://github.com/DDVD233/QoQ_Med.