Cross-modal knowledge distillation for action recognition
This addresses the challenge of modality adaptation in action recognition, offering a practical solution for scenarios with limited labeled data, though it is incremental in its approach.
The paper tackles the problem of adapting an action recognition network trained on one modality (e.g., RGB videos) to another modality (e.g., 3D human poses) without annotated data, achieving accuracy nearly equivalent to fully supervised training.
In this work, we address the problem how a network for action recognition that has been trained on a modality like RGB videos can be adapted to recognize actions for another modality like sequences of 3D human poses. To this end, we extract the knowledge of the trained teacher network for the source modality and transfer it to a small ensemble of student networks for the target modality. For the cross-modal knowledge distillation, we do not require any annotated data. Instead we use pairs of sequences of both modalities as supervision, which are straightforward to acquire. In contrast to previous works for knowledge distillation that use a KL-loss, we show that the cross-entropy loss together with mutual learning of a small ensemble of student networks performs better. In fact, the proposed approach for cross-modal knowledge distillation nearly achieves the accuracy of a student network trained with full supervision.