Qffusion: Controllable Portrait Video Editing via Quadrant-Grid Attention Learning
This work addresses portrait video editing for users needing controllable and stable video generation, representing an incremental improvement by building on existing Stable Diffusion methods.
The paper tackles portrait video editing by introducing Qffusion, a dual-frame-guided framework that uses a Quadrant-grid Arrangement scheme and Stable Diffusion to achieve stable video editing without extra networks, consistently outperforming state-of-the-art techniques.
This paper presents Qffusion, a dual-frame-guided framework for portrait video editing. Specifically, we consider a design principle of ``animation for editing'', and train Qffusion as a general animation framework from two still reference images while we can use it for portrait video editing easily by applying modified start and end frames as references during inference. Leveraging the powerful generative power of Stable Diffusion, we propose a Quadrant-grid Arrangement (QGA) scheme for latent re-arrangement, which arranges the latent codes of two reference images and that of four facial conditions into a four-grid fashion, separately. Then, we fuse features of these two modalities and use self-attention for both appearance and temporal learning, where representations at different times are jointly modeled under QGA. Our Qffusion can achieve stable video editing without additional networks or complex training stages, where only the input format of Stable Diffusion is modified. Further, we propose a Quadrant-grid Propagation (QGP) inference strategy, which enjoys a unique advantage on stable arbitrary-length video generation by processing reference and condition frames recursively. Through extensive experiments, Qffusion consistently outperforms state-of-the-art techniques on portrait video editing. Project page: https://qffusion.github.io/page/.