Capture Stage Matting: Challenges, Approaches, and Solutions for Offline and Real-Time Processing
This work provides practical guidelines for practitioners in media production to improve matting workflows for high-end capture stages, though it is incremental as it adapts existing methods rather than introducing new ones.
The paper addresses the challenge of applying matting algorithms to capture stage content, where existing methods struggle due to specific characteristics, and presents an efficient pipeline for adapting state-of-the-art approaches to offline and real-time processing without extensive annotations.
Capture stages are high-end sources of state-of-the-art recordings for downstream applications in movies, games, and other media. One crucial step in almost all pipelines is matting, i.e., separating captured performances from the background. While common matting algorithms deliver remarkable performance in other applications like teleconferencing and mobile entertainment, we found that they struggle significantly with the peculiarities of capture stage content. The goal of our work is to share insights into those challenges as a curated list of these characteristics along with a constructive discussion for proactive intervention and present a guideline to practitioners for an improved workflow to mitigate unresolved challenges. To this end, we also demonstrate an efficient pipeline to adapt state-of-the-art approaches to such custom setups without the need for extensive annotations, both offline and real-time. For an objective evaluation, we introduce a validation methodology using a state-of-the-art diffusion model to demonstrate the benefits of our approach.