A Temporally-Aware Interpolation Network for Video Frame Inpainting
This addresses video editing, manipulation, and forensics by providing a less ambiguous task than frame interpolation or prediction, though it appears incremental as it builds on existing video prediction and inpainting techniques.
The authors tackled video frame inpainting by proposing a deep learning pipeline with bidirectional prediction and temporally-aware interpolation, achieving more accurate and qualitatively satisfying results compared to state-of-the-art methods.
We propose the first deep learning solution to video frame inpainting, a challenging instance of the general video inpainting problem with applications in video editing, manipulation, and forensics. Our task is less ambiguous than frame interpolation and video prediction because we have access to both the temporal context and a partial glimpse of the future, allowing us to better evaluate the quality of a model's predictions objectively. We devise a pipeline composed of two modules: a bidirectional video prediction module, and a temporally-aware frame interpolation module. The prediction module makes two intermediate predictions of the missing frames, one conditioned on the preceding frames and the other conditioned on the following frames, using a shared convolutional LSTM-based encoder-decoder. The interpolation module blends the intermediate predictions to form the final result. Specifically, it utilizes time information and hidden activations from the video prediction module to resolve disagreements between the predictions. Our experiments demonstrate that our approach produces more accurate and qualitatively satisfying results than a state-of-the-art video prediction method and many strong frame inpainting baselines.