VASE: Object-Centric Appearance and Shape Manipulation of Real Videos
This work addresses the need for more detailed and explicit object manipulation in video editing, offering incremental improvements over existing holistic editing approaches.
The paper tackles the problem of object-centric video editing by enabling precise control over both appearance and structural modifications of objects, achieving similar performance to state-of-the-art methods in image-driven video editing while introducing novel shape-editing capabilities.
Recently, several works tackled the video editing task fostered by the success of large-scale text-to-image generative models. However, most of these methods holistically edit the frame using the text, exploiting the prior given by foundation diffusion models and focusing on improving the temporal consistency across frames. In this work, we introduce a framework that is object-centric and is designed to control both the object's appearance and, notably, to execute precise and explicit structural modifications on the object. We build our framework on a pre-trained image-conditioned diffusion model, integrate layers to handle the temporal dimension, and propose training strategies and architectural modifications to enable shape control. We evaluate our method on the image-driven video editing task showing similar performance to the state-of-the-art, and showcasing novel shape-editing capabilities. Further details, code and examples are available on our project page: https://helia95.github.io/vase-website/