Agglomerating Large Vision Encoders via Distillation for VFSS Segmentation
This work addresses the efficiency-performance trade-off for medical image segmentation, offering a domain-specific incremental improvement.
The paper tackled the high training and inference complexity of large medical foundation models for segmentation by proposing a knowledge distillation framework from multiple specialized models, achieving a 2% average Dice coefficient gain over simple distillation.
The deployment of foundation models for medical imaging has demonstrated considerable success. However, their training overheads associated with downstream tasks remain substantial due to the size of the image encoders employed, and the inference complexity is also significantly high. Although lightweight variants have been obtained for these foundation models, their performance is constrained by their limited model capacity and suboptimal training strategies. In order to achieve an improved tradeoff between complexity and performance, we propose a new framework to improve the performance of low complexity models via knowledge distillation from multiple large medical foundation models (e.g., MedSAM, RAD-DINO, MedCLIP), each specializing in different vision tasks, with the goal to effectively bridge the performance gap for medical image segmentation tasks. The agglomerated model demonstrates superior generalization across 12 segmentation tasks, whereas specialized models require explicit training for each task. Our approach achieved an average performance gain of 2\% in Dice coefficient compared to simple distillation.