Asymmetric Cross-Modal Knowledge Distillation: Bridging Modalities with Weak Semantic Consistency
This work addresses a practical limitation in real-world cross-modal learning for remote sensing scene classification, though it appears incremental by extending existing distillation methods to asymmetric scenarios.
The paper tackles the problem of cross-modal knowledge distillation when modalities have weak semantic consistency, proposing the Asymmetric Cross-modal Knowledge Distillation (ACKD) concept and a framework called SemBridge to address challenges in knowledge transmission. The result is state-of-the-art performance compared to 7 existing approaches on 6 model architectures across various datasets.
Cross-modal Knowledge Distillation has demonstrated promising performance on paired modalities with strong semantic connections, referred to as Symmetric Cross-modal Knowledge Distillation (SCKD). However, implementing SCKD becomes exceedingly constrained in real-world scenarios due to the limited availability of paired modalities. To this end, we investigate a general and effective knowledge learning concept under weak semantic consistency, dubbed Asymmetric Cross-modal Knowledge Distillation (ACKD), aiming to bridge modalities with limited semantic overlap. Nevertheless, the shift from strong to weak semantic consistency improves flexibility but exacerbates challenges in knowledge transmission costs, which we rigorously verified based on optimal transport theory. To mitigate the issue, we further propose a framework, namely SemBridge, integrating a Student-Friendly Matching module and a Semantic-aware Knowledge Alignment module. The former leverages self-supervised learning to acquire semantic-based knowledge and provide personalized instruction for each student sample by dynamically selecting the relevant teacher samples. The latter seeks the optimal transport path by employing Lagrangian optimization. To facilitate the research, we curate a benchmark dataset derived from two modalities, namely Multi-Spectral (MS) and asymmetric RGB images, tailored for remote sensing scene classification. Comprehensive experiments exhibit that our framework achieves state-of-the-art performance compared with 7 existing approaches on 6 different model architectures across various datasets.