MedMoE: Modality-Specialized Mixture of Experts for Medical Vision-Language Understanding
This addresses the need for better medical vision-language systems by incorporating modality-specific visual representations, though it appears incremental as it builds on existing MoE and transformer methods.
The paper tackles the problem of medical vision-language understanding by proposing MedMoE, a framework that dynamically adapts visual representation based on diagnostic context using a Mixture-of-Experts module, which improves alignment and retrieval performance across imaging modalities.
Different medical imaging modalities capture diagnostic information at varying spatial resolutions, from coarse global patterns to fine-grained localized structures. However, most existing vision-language frameworks in the medical domain apply a uniform strategy for local feature extraction, overlooking the modality-specific demands. In this work, we present MedMoE, a modular and extensible vision-language processing framework that dynamically adapts visual representation based on the diagnostic context. MedMoE incorporates a Mixture-of-Experts (MoE) module conditioned on the report type, which routes multi-scale image features through specialized expert branches trained to capture modality-specific visual semantics. These experts operate over feature pyramids derived from a Swin Transformer backbone, enabling spatially adaptive attention to clinically relevant regions. This framework produces localized visual representations aligned with textual descriptions, without requiring modality-specific supervision at inference. Empirical results on diverse medical benchmarks demonstrate that MedMoE improves alignment and retrieval performance across imaging modalities, underscoring the value of modality-specialized visual representations in clinical vision-language systems.