IKD+: Reliable Low Complexity Deep Models For Retinopathy Classification
This work addresses the need for low-complexity and reliable models for retinopathy classification, which is incremental as it builds on existing knowledge distillation techniques.
The paper tackles the problem of high complexity and unreliable predictions in deep neural network models for retinopathy classification by proposing IKD+, an iterative knowledge distillation method that balances size, accuracy, and reliability, achieving up to 500-fold parameter reduction without significant accuracy loss.
Deep neural network (DNN) models for retinopathy have estimated predictive accuracies in the mid-to-high 90%. However, the following aspects remain unaddressed: State-of-the-art models are complex and require substantial computational infrastructure to train and deploy; The reliability of predictions can vary widely. In this paper, we focus on these aspects and propose a form of iterative knowledge distillation(IKD), called IKD+ that incorporates a tradeoff between size, accuracy and reliability. We investigate the functioning of IKD+ using two widely used techniques for estimating model calibration (Platt-scaling and temperature-scaling), using the best-performing model available, which is an ensemble of EfficientNets with approximately 100M parameters. We demonstrate that IKD+ equipped with temperature-scaling results in models that show up to approximately 500-fold decreases in the number of parameters than the original ensemble without a significant loss in accuracy. In addition, calibration scores (reliability) for the IKD+ models are as good as or better than the base mode