Castle in the Sky: Dynamic Sky Replacement and Harmonization in Videos
This addresses the need for accessible video sky editing for creators, offering a purely vision-based method that works with common devices like smartphones and dash cameras, though it is incremental as it builds on existing sky editing techniques.
The paper tackles the problem of automatically replacing and harmonizing sky backgrounds in videos to generate realistic and dramatic results with controllable styles, achieving real-time performance without user interactions and demonstrating high fidelity and good generalization in experiments.
This paper proposes a vision-based method for video sky replacement and harmonization, which can automatically generate realistic and dramatic sky backgrounds in videos with controllable styles. Different from previous sky editing methods that either focus on static photos or require inertial measurement units integrated in smartphones on shooting videos, our method is purely vision-based, without any requirements on the capturing devices, and can be well applied to either online or offline processing scenarios. Our method runs in real-time and is free of user interactions. We decompose this artistic creation process into a couple of proxy tasks including sky matting, motion estimation, and image blending. Experiments are conducted on videos diversely captured in the wild by handheld smartphones and dash cameras, and show high fidelity and good generalization of our method in both visual quality and lighting/motion dynamics. Our code and animated results are available at \url{https://jiupinjia.github.io/skyar/}.