ANYPORTAL: Zero-Shot Consistent Video Background Replacement
This addresses the challenge of precise video editing for content creators, though it appears incremental as it builds on existing diffusion models.
The paper tackles the problem of achieving fine-grained control in video background replacement by introducing ANYPORTAL, a zero-shot framework that ensures foreground consistency and temporally coherent relighting, resulting in high-quality outputs on consumer-grade GPUs.
Despite the rapid advancements in video generation technology, creating high-quality videos that precisely align with user intentions remains a significant challenge. Existing methods often fail to achieve fine-grained control over video details, limiting their practical applicability. We introduce ANYPORTAL, a novel zero-shot framework for video background replacement that leverages pre-trained diffusion models. Our framework collaboratively integrates the temporal prior of video diffusion models with the relighting capabilities of image diffusion models in a zero-shot setting. To address the critical challenge of foreground consistency, we propose a Refinement Projection Algorithm, which enables pixel-level detail manipulation to ensure precise foreground preservation. ANYPORTAL is training-free and overcomes the challenges of achieving foreground consistency and temporally coherent relighting. Experimental results demonstrate that ANYPORTAL achieves high-quality results on consumer-grade GPUs, offering a practical and efficient solution for video content creation and editing.