Comprehensive Video Understanding: Video summarization with content-based video recommender design
This work addresses video summarization for users overwhelmed by long videos, presenting an incremental approach by integrating recommender systems and multi-task learning.
The paper tackles video summarization by framing it as a content-based recommender problem to distill useful content for users with information overload, achieving first place in the ICCV 2019 CoView Workshop Challenge Track.
Video summarization aims to extract keyframes/shots from a long video. Previous methods mainly take diversity and representativeness of generated summaries as prior knowledge in algorithm design. In this paper, we formulate video summarization as a content-based recommender problem, which should distill the most useful content from a long video for users who suffer from information overload. A scalable deep neural network is proposed on predicting if one video segment is a useful segment for users by explicitly modelling both segment and video. Moreover, we accomplish scene and action recognition in untrimmed videos in order to find more correlations among different aspects of video understanding tasks. Also, our paper will discuss the effect of audio and visual features in summarization task. We also extend our work by data augmentation and multi-task learning for preventing the model from early-stage overfitting. The final results of our model win the first place in ICCV 2019 CoView Workshop Challenge Track.