Distilling Audio-Visual Knowledge by Compositional Contrastive Learning
This work addresses the challenge of multi-modal knowledge distillation for video representation learning, offering a novel approach that is incremental in improving existing methods.
The paper tackles the problem of transferring knowledge across heterogeneous audio-visual modalities by proposing compositional contrastive learning to close the cross-modal semantic gap, resulting in significant performance improvements over existing knowledge distillation methods on video datasets like UCF101, ActivityNet, and VGGSound.
Having access to multi-modal cues (e.g. vision and audio) empowers some cognitive tasks to be done faster compared to learning from a single modality. In this work, we propose to transfer knowledge across heterogeneous modalities, even though these data modalities may not be semantically correlated. Rather than directly aligning the representations of different modalities, we compose audio, image, and video representations across modalities to uncover richer multi-modal knowledge. Our main idea is to learn a compositional embedding that closes the cross-modal semantic gap and captures the task-relevant semantics, which facilitates pulling together representations across modalities by compositional contrastive learning. We establish a new, comprehensive multi-modal distillation benchmark on three video datasets: UCF101, ActivityNet, and VGGSound. Moreover, we demonstrate that our model significantly outperforms a variety of existing knowledge distillation methods in transferring audio-visual knowledge to improve video representation learning. Code is released here: https://github.com/yanbeic/CCL.