V-LASIK: Consistent Glasses-Removal from Videos Using Synthetic Data
This addresses the challenge of consistent local attribute removal in videos for video editing applications, representing an incremental advance in leveraging synthetic data and priors.
The paper tackled the problem of consistently removing glasses from videos while preserving identity, using synthetic data and a pretrained diffusion model, and demonstrated significant improvement over existing methods with successful generalization to other tasks like facial sticker-removal.
Diffusion-based generative models have recently shown remarkable image and video editing capabilities. However, local video editing, particularly removal of small attributes like glasses, remains a challenge. Existing methods either alter the videos excessively, generate unrealistic artifacts, or fail to perform the requested edit consistently throughout the video. In this work, we focus on consistent and identity-preserving removal of glasses in videos, using it as a case study for consistent local attribute removal in videos. Due to the lack of paired data, we adopt a weakly supervised approach and generate synthetic imperfect data, using an adjusted pretrained diffusion model. We show that despite data imperfection, by learning from our generated data and leveraging the prior of pretrained diffusion models, our model is able to perform the desired edit consistently while preserving the original video content. Furthermore, we exemplify the generalization ability of our method to other local video editing tasks by applying it successfully to facial sticker-removal. Our approach demonstrates significant improvement over existing methods, showcasing the potential of leveraging synthetic data and strong video priors for local video editing tasks.