Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles
This work addresses the challenge of high compute costs in ensemble models for respiratory sound classification, offering an incremental improvement with practical benefits for medical diagnostics.
The paper tackled the problem of limited respiratory sound datasets by using architecture-agnostic knowledge distillation from ensembles to improve classification performance, achieving a new state-of-the-art score of 64.39 on ICHBI, surpassing the previous best by 0.85 and improving average scores by over 1.16.
Respiratory sound datasets are limited in size and quality, making high performance difficult to achieve. Ensemble models help but inevitably increase compute cost at inference time. Soft label training distills knowledge efficiently with extra cost only at training. In this study, we explore soft labels for respiratory sound classification as an architecture-agnostic approach to distill an ensemble of teacher models into a student model. We examine different variations of our approach and find that even a single teacher, identical to the student, considerably improves performance beyond its own capability, with optimal gains achieved using only a few teachers. We achieve the new state-of-the-art Score of 64.39 on ICHBI, surpassing the previous best by 0.85 and improving average Scores across architectures by more than 1.16. Our results highlight the effectiveness of knowledge distillation with soft labels for respiratory sound classification, regardless of size or architecture.