On Compressing U-net Using Knowledge Distillation
This work addresses model compression for U-net, which is widely used in medical imaging and segmentation tasks, offering a practical solution for deployment in resource-constrained environments.
The paper tackles the problem of compressing U-net architectures using knowledge distillation, achieving over 1000x compression to 0.1% of original parameters with negligible performance loss by incorporating regularization methods like batch normalization and class re-weighting.
We study the use of knowledge distillation to compress the U-net architecture. We show that, while standard distillation is not sufficient to reliably train a compressed U-net, introducing other regularization methods, such as batch normalization and class re-weighting, in knowledge distillation significantly improves the training process. This allows us to compress a U-net by over 1000x, i.e., to 0.1% of its original number of parameters, at a negligible decrease in performance.