WeCKD: Weakly-supervised Chained Distillation Network for Efficient Multimodal Medical Imaging
This addresses the need for efficient multimodal medical imaging with minimal supervision, though it appears incremental as it builds on existing knowledge distillation methods.
The paper tackled the problem of knowledge degradation and inefficient supervision in knowledge distillation by introducing a weakly-supervised chain-based distillation network, achieving cumulative accuracy gains of up to +23% over a single backbone model on limited data.
Knowledge distillation (KD) has traditionally relied on a static teacher-student framework, where a large, well-trained teacher transfers knowledge to a single student model. However, these approaches often suffer from knowledge degradation, inefficient supervision, and reliance on either a very strong teacher model or large labeled datasets. To address these, we present the first-ever Weakly-supervised Chain-based KD network (WeCKD) that redefines knowledge transfer through a structured sequence of interconnected models. Unlike conventional KD, it forms a progressive distillation chain, where each model not only learns from its predecessor but also refines the knowledge before passing it forward. This structured knowledge transfer further enhances feature learning and addresses the limitations of one-step KD. Each model in the chain is trained on only a fraction of the dataset and shows that effective learning can be achieved with minimal supervision. Extensive evaluation on six imaging datasets across otoscopic, microscopic, and magnetic resonance imaging modalities shows that it generalizes and outperforms existing methods. Furthermore, the proposed distillation chain resulted in cumulative accuracy gains of up to +23% over a single backbone trained on the same limited data, which highlights its potential for real-world adoption.