Leveraging Topological Guidance for Improved Knowledge Distillation
This work addresses the problem of deploying complex models on resource-constrained devices, though it appears incremental as it builds on existing knowledge distillation and topological data analysis methods.
The paper tackles the challenge of high computational cost in extracting topological features for deep learning by proposing a knowledge distillation framework that uses topological guidance to train lightweight models for image classification, achieving improved performance.
Deep learning has shown its efficacy in extracting useful features to solve various computer vision tasks. However, when the structure of the data is complex and noisy, capturing effective information to improve performance is very difficult. To this end, topological data analysis (TDA) has been utilized to derive useful representations that can contribute to improving performance and robustness against perturbations. Despite its effectiveness, the requirements for large computational resources and significant time consumption in extracting topological features through TDA are critical problems when implementing it on small devices. To address this issue, we propose a framework called Topological Guidance-based Knowledge Distillation (TGD), which uses topological features in knowledge distillation (KD) for image classification tasks. We utilize KD to train a superior lightweight model and provide topological features with multiple teachers simultaneously. We introduce a mechanism for integrating features from different teachers and reducing the knowledge gap between teachers and the student, which aids in improving performance. We demonstrate the effectiveness of our approach through diverse empirical evaluations.