Learning Video-Story Composition via Recurrent Neural Network
This addresses the challenge of creating coherent video-stories from clips for applications like video editing or storytelling, but it appears incremental as it builds on existing methods with optimizations.
The paper tackles the problem of composing a video-story from video clips by learning coherence using a Recurrent Neural Network (RNN) that incorporates spatial-temporal semantics and motion dynamics, and it outperforms the state-of-the-art approach on a video-story dataset.
In this paper, we propose a learning-based method to compose a video-story from a group of video clips that describe an activity or experience. We learn the coherence between video clips from real videos via the Recurrent Neural Network (RNN) that jointly incorporates the spatial-temporal semantics and motion dynamics to generate smooth and relevant compositions. We further rearrange the results generated by the RNN to make the overall video-story compatible with the storyline structure via a submodular ranking optimization process. Experimental results on the video-story dataset show that the proposed algorithm outperforms the state-of-the-art approach.