Preserve Pre-trained Knowledge: Transfer Learning With Self-Distillation For Action Recognition
This addresses a specific problem of knowledge loss in action recognition for computer vision researchers, but it is incremental as it builds on existing transfer learning frameworks.
The paper tackles catastrophic forgetting in transfer learning for video action recognition by proposing a self-distillation method during fine-tuning to preserve pre-trained knowledge, achieving state-of-the-art results on UCF101 and HMDB51 datasets.
Video-based action recognition is one of the most popular topics in computer vision. With recent advances of selfsupervised video representation learning approaches, action recognition usually follows a two-stage training framework, i.e., self-supervised pre-training on large-scale unlabeled sets and transfer learning on a downstream labeled set. However, catastrophic forgetting of the pre-trained knowledge becomes the main issue in the downstream transfer learning of action recognition, resulting in a sub-optimal solution. In this paper, to alleviate the above issue, we propose a novel transfer learning approach that combines self-distillation in fine-tuning to preserve knowledge from the pre-trained model learned from the large-scale dataset. Specifically, we fix the encoder from the last epoch as the teacher model to guide the training of the encoder from the current epoch in the transfer learning. With such a simple yet effective learning strategy, we outperform state-of-the-art methods on widely used UCF101 and HMDB51 datasets in action recognition task.