Cross-modal Contrastive Distillation for Instructional Activity Anticipation
This work addresses the challenge of instructional activity anticipation for video understanding, offering interpretable outputs, but it is incremental as it builds on knowledge distillation with a novel cross-modal adaptation.
The paper tackles the problem of generating natural language descriptions of future action steps from instructional videos, proposing a cross-modal contrastive distillation framework that improves the visual-alone model's anticipation performance by 40.2% in BLEU4 on the Tasty Videos dataset.
In this study, we aim to predict the plausible future action steps given an observation of the past and study the task of instructional activity anticipation. Unlike previous anticipation tasks that aim at action label prediction, our work targets at generating natural language outputs that provide interpretable and accurate descriptions of future action steps. It is a challenging task due to the lack of semantic information extracted from the instructional videos. To overcome this challenge, we propose a novel knowledge distillation framework to exploit the related external textual knowledge to assist the visual anticipation task. However, previous knowledge distillation techniques generally transfer information within the same modality. To bridge the gap between the visual and text modalities during the distillation process, we devise a novel cross-modal contrastive distillation (CCD) scheme, which facilitates knowledge distillation between teacher and student in heterogeneous modalities with the proposed cross-modal distillation loss. We evaluate our method on the Tasty Videos dataset. CCD improves the anticipation performance of the visual-alone student model by a large margin of 40.2% relatively in BLEU4. Our approach also outperforms the state-of-the-art approaches by a large margin.