I Have Seen Enough: A Teacher Student Network for Video Classification Using Fewer Frames
This addresses efficiency issues for video analysis tasks, but it is incremental as it builds on existing distillation methods.
The paper tackles the problem of high computational time in video classification by proposing a teacher-student network that uses distillation to reduce the number of frames processed, achieving up to 30% faster inference with minimal performance drop on the YouTube-8M dataset.
Over the past few years, various tasks involving videos such as classification, description, summarization and question answering have received a lot of attention. Current models for these tasks compute an encoding of the video by treating it as a sequence of images and going over every image in the sequence. However, for longer videos this is very time consuming. In this paper, we focus on the task of video classification and aim to reduce the computational time by using the idea of distillation. Specifically, we first train a teacher network which looks at all the frames in a video and computes a representation for the video. We then train a student network whose objective is to process only a small fraction of the frames in the video and still produce a representation which is very close to the representation computed by the teacher network. This smaller student network involving fewer computations can then be employed at inference time for video classification. We experiment with the YouTube-8M dataset and show that the proposed student network can reduce the inference time by upto 30% with a very small drop in the performance