Frame-Recurrent Video Inpainting by Robust Optical Flow Inference
This addresses the problem of efficiently and consistently filling missing regions in videos for applications like video editing and restoration, representing a strong specific gain in the domain.
The paper tackles video inpainting by proposing a deep learning architecture that uses ConvLSTM and optical flow to preserve temporal consistency and spatial details, enabling real-time processing of arbitrary video sizes and lengths with superior performance compared to state-of-the-art methods.
In this paper, we present a new inpainting framework for recovering missing regions of video frames. Compared with image inpainting, performing this task on video presents new challenges such as how to preserving temporal consistency and spatial details, as well as how to handle arbitrary input video size and length fast and efficiently. Towards this end, we propose a novel deep learning architecture which incorporates ConvLSTM and optical flow for modeling the spatial-temporal consistency in videos. It also saves much computational resource such that our method can handle videos with larger frame size and arbitrary length streamingly in real-time. Furthermore, to generate an accurate optical flow from corrupted frames, we propose a robust flow generation module, where two sources of flows are fed and a flow blending network is trained to fuse them. We conduct extensive experiments to evaluate our method in various scenarios and different datasets, both qualitatively and quantitatively. The experimental results demonstrate the superior of our method compared with the state-of-the-art inpainting approaches.