Dual-Model Weight Selection and Self-Knowledge Distillation for Medical Image Classification
This work addresses computational efficiency challenges for medical practitioners in deploying AI models, but it is incremental as it builds on existing knowledge distillation and lightweight model techniques.
The authors tackled the problem of deploying large-scale models in medical image classification under computational constraints by proposing a method that integrates dual-model weight selection with self-knowledge distillation, achieving superior performance and robustness on datasets like chest X-rays, lung CT scans, and brain MRI scans compared to existing methods.
We propose a novel medical image classification method that integrates dual-model weight selection with self-knowledge distillation (SKD). In real-world medical settings, deploying large-scale models is often limited by computational resource constraints, which pose significant challenges for their practical implementation. Thus, developing lightweight models that achieve comparable performance to large-scale models while maintaining computational efficiency is crucial. To address this, we employ a dual-model weight selection strategy that initializes two lightweight models with weights derived from a large pretrained model, enabling effective knowledge transfer. Next, SKD is applied to these selected models, allowing the use of a broad range of initial weight configurations without imposing additional excessive computational cost, followed by fine-tuning for the target classification tasks. By combining dual-model weight selection with self-knowledge distillation, our method overcomes the limitations of conventional approaches, which often fail to retain critical information in compact models. Extensive experiments on publicly available datasets-chest X-ray images, lung computed tomography scans, and brain magnetic resonance imaging scans-demonstrate the superior performance and robustness of our approach compared to existing methods.