TIE-KD: Teacher-Independent and Explainable Knowledge Distillation for Monocular Depth Estimation
This addresses the need for efficient and interpretable models in applications like autonomous driving, though it is incremental as it builds on existing knowledge distillation techniques.
The paper tackled the problem of high computational demands in monocular depth estimation by introducing TIE-KD, a teacher-independent and explainable knowledge distillation framework that uses Depth Probability Maps to transfer knowledge without requiring architectural similarity, resulting in outperforming conventional methods on the KITTI dataset.
Monocular depth estimation (MDE) is essential for numerous applications yet is impeded by the substantial computational demands of accurate deep learning models. To mitigate this, we introduce a novel Teacher-Independent Explainable Knowledge Distillation (TIE-KD) framework that streamlines the knowledge transfer from complex teacher models to compact student networks, eliminating the need for architectural similarity. The cornerstone of TIE-KD is the Depth Probability Map (DPM), an explainable feature map that interprets the teacher's output, enabling feature-based knowledge distillation solely from the teacher's response. This approach allows for efficient student learning, leveraging the strengths of feature-based distillation. Extensive evaluation of the KITTI dataset indicates that TIE-KD not only outperforms conventional response-based KD methods but also demonstrates consistent efficacy across diverse teacher and student architectures. The robustness and adaptability of TIE-KD underscore its potential for applications requiring efficient and interpretable models, affirming its practicality for real-world deployment.