Over++: Generative Video Compositing for Layer Interaction Effects
This addresses a domain-specific problem for professional video artists by automating environmental interaction effects, though it is incremental as it builds on existing generative models.
The paper tackles the problem of manually creating environmental interactions in video compositing by introducing Over++, a framework that synthesizes realistic effects like shadows and reflections while preserving the original video, outperforming existing baselines in effect generation and scene preservation.
In professional video compositing workflows, artists must manually create environmental interactions-such as shadows, reflections, dust, and splashes-between foreground subjects and background layers. Existing video generative models struggle to preserve the input video while adding such effects, and current video inpainting methods either require costly per-frame masks or yield implausible results. We introduce augmented compositing, a new task that synthesizes realistic, semi-transparent environmental effects conditioned on text prompts and input video layers, while preserving the original scene. To address this task, we present Over++, a video effect generation framework that makes no assumptions about camera pose, scene stationarity, or depth supervision. We construct a paired effect dataset tailored for this task and introduce an unpaired augmentation strategy that preserves text-driven editability. Our method also supports optional mask control and keyframe guidance without requiring dense annotations. Despite training on limited data, Over++ produces diverse and realistic environmental effects and outperforms existing baselines in both effect generation and scene preservation.