VORNet: Spatio-temporally Consistent Video Inpainting for Object Removal
This work addresses the challenge of video object removal for video processing applications, offering a more consistent solution, though it appears incremental as it builds on existing image inpainting techniques.
The paper tackles the problem of video object removal, where existing deep learning methods often produce inconsistent results between frames, by proposing VORNet, a learning-based network that combines optical flow warping and image-based inpainting to achieve spatio-temporally consistent video inpainting, with experiments on a synthesized dataset showing improved consistency over existing methods.
Video object removal is a challenging task in video processing that often requires massive human efforts. Given the mask of the foreground object in each frame, the goal is to complete (inpaint) the object region and generate a video without the target object. While recently deep learning based methods have achieved great success on the image inpainting task, they often lead to inconsistent results between frames when applied to videos. In this work, we propose a novel learning-based Video Object Removal Network (VORNet) to solve the video object removal task in a spatio-temporally consistent manner, by combining the optical flow warping and image-based inpainting model. Experiments are done on our Synthesized Video Object Removal (SVOR) dataset based on the YouTube-VOS video segmentation dataset, and both the objective and subjective evaluation demonstrate that our VORNet generates more spatially and temporally consistent videos compared with existing methods.