Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production
This addresses the problem of simplifying video pre-production for amateurs, though it is incremental as it builds on existing simulation engines and ranking methods.
The paper tackles the time-consuming process of setting scenes and cameras for amateur video production by introducing Virtual Dynamic Storyboard (VDS), which generates and ranks dynamic storyboards in a virtual environment, validated through experiments and user studies to show efficiency and effectiveness.
Amateurs working on mini-films and short-form videos usually spend lots of time and effort on the multi-round complicated process of setting and adjusting scenes, plots, and cameras to deliver satisfying video shots. We present Virtual Dynamic Storyboard (VDS) to allow users storyboarding shots in virtual environments, where the filming staff can easily test the settings of shots before the actual filming. VDS runs on a "propose-simulate-discriminate" mode: Given a formatted story script and a camera script as input, it generates several character animation and camera movement proposals following predefined story and cinematic rules to allow an off-the-shelf simulation engine to render videos. To pick up the top-quality dynamic storyboard from the candidates, we equip it with a shot ranking discriminator based on shot quality criteria learned from professional manual-created data. VDS is comprehensively validated via extensive experiments and user studies, demonstrating its efficiency, effectiveness, and great potential in assisting amateur video production.