Learnable Instance Attention Filtering for Adaptive Detector Distillation
This work addresses deployment efficiency for vision models by improving knowledge distillation, though it is incremental as it builds on existing feature-based methods.
The paper tackled the problem of inefficient deployment of complex deep vision models by proposing a learnable instance attention filtering method for knowledge distillation, which achieved a 2% performance gain on a GFL ResNet-50 student model without added complexity.
As deep vision models grow increasingly complex to achieve higher performance, deployment efficiency has become a critical concern. Knowledge distillation (KD) mitigates this issue by transferring knowledge from large teacher models to compact student models. While many feature-based KD methods rely on spatial filtering to guide distillation, they typically treat all object instances uniformly, ignoring instance-level variability. Moreover, existing attention filtering mechanisms are typically heuristic or teacher-driven, rather than learned with the student. To address these limitations, we propose Learnable Instance Attention Filtering for Adaptive Detector Distillation (LIAF-KD), a novel framework that introduces learnable instance selectors to dynamically evaluate and reweight instance importance during distillation. Notably, the student contributes to this process based on its evolving learning state. Experiments on the KITTI and COCO datasets demonstrate consistent improvements, with a 2% gain on a GFL ResNet-50 student without added complexity, outperforming state-of-the-art methods.