Sparse Spectral LoRA: Routed Experts for Medical VLMs
This addresses robustness and efficiency issues for medical AI applications, though it is incremental as it builds on existing parameter-efficient fine-tuning and mixture-of-experts methods.
The paper tackled the problem of large vision-language models lacking robustness in medical imaging due to cross-dataset interference and catastrophic forgetting in sequential tasks, resulting in MedQwen achieving near full fine-tuning performance with 339x fewer parameters and reducing forgetting to ~5% compared to baselines degrading by >20-50%.
Large vision-language models (VLMs) excel on general benchmarks but often lack robustness in medical imaging, where heterogeneous supervision induces cross-dataset interference and sensitivity to data regime (i.e., how the supervisory signals are mixed). In realistic clinical workflows, data and tasks arrive sequentially, so naive continual training further leads to catastrophic forgetting. To address these challenges, we propose MedQwen, a parameter-efficient medical VLM that couples a spectrally routed Mixture-of-Experts (MoE) with a theoretically grounded scaling rule that aligns low-rank updates with a full-rank, fully fine-tuned MoE, without changing the base architecture. Concretely, we initialize each expert from non-overlapping singular value decomposition (SVD) segments of the pretrained weight and introduce a residual compensation and scaling scheme to enable stable expert specialization and consistent routing under distribution shift. Across 23 medical datasets covering visual question answering, report generation, radiology classification, and hallucination mitigation, MedQwen achieves strong, reliable performance: it approaches full fine-tuning on zero-shot classification with 339$\times$ fewer trainable parameters, and reduces sequential forgetting to $\sim$5\% where strong baselines degrade by $>$20-50\%.