Enhanced Multimodal Representation Learning with Cross-modal KD
This work addresses a specific bottleneck in multimodal representation learning for tasks like video analysis and emotion classification, offering incremental improvements over existing methods.
The paper tackles the problem of weak teacher solutions in cross-modal knowledge distillation by introducing mutual information between teacher and auxiliary modality models and minimizing conditional entropy of teacher given student, achieving state-of-the-art results on video recognition, retrieval, and emotion classification benchmarks.
This paper explores the tasks of leveraging auxiliary modalities which are only available at training to enhance multimodal representation learning through cross-modal Knowledge Distillation (KD). The widely adopted mutual information maximization-based objective leads to a short-cut solution of the weak teacher, i.e., achieving the maximum mutual information by simply making the teacher model as weak as the student model. To prevent such a weak solution, we introduce an additional objective term, i.e., the mutual information between the teacher and the auxiliary modality model. Besides, to narrow down the information gap between the student and teacher, we further propose to minimize the conditional entropy of the teacher given the student. Novel training schemes based on contrastive learning and adversarial learning are designed to optimize the mutual information and the conditional entropy, respectively. Experimental results on three popular multimodal benchmark datasets have shown that the proposed method outperforms a range of state-of-the-art approaches for video recognition, video retrieval and emotion classification.