Automated Video Segmentation Machine Learning Pipeline
This work addresses the specific problem of mask generation for visual effects artists, but it appears incremental as it combines existing methods like object detection, image segmentation, and video tracking into a pipeline.
The paper tackled the problem of slow and resource-intensive mask generation in visual effects production by presenting an automated video segmentation pipeline that creates temporally consistent instance masks, resulting in significantly reduced manual effort and faster creation of preliminary composites.
Visual effects (VFX) production often struggles with slow, resource-intensive mask generation. This paper presents an automated video segmentation pipeline that creates temporally consistent instance masks. It employs machine learning for: (1) flexible object detection via text prompts, (2) refined per-frame image segmentation and (3) robust video tracking to ensure temporal stability. Deployed using containerization and leveraging a structured output format, the pipeline was quickly adopted by our artists. It significantly reduces manual effort, speeds up the creation of preliminary composites, and provides comprehensive segmentation data, thereby enhancing overall VFX production efficiency.