Video-ReTime: Learning Temporally Varying Speediness for Time Remapping
This work addresses video editing and processing needs for applications like film production or content creation, offering incremental improvements in speed and robustness.
The paper tackles the problem of generating temporally remapped videos to match target durations while preserving natural dynamics, achieving more accurate detection of playback speed variations and greater efficiency than prior methods.
We propose a method for generating a temporally remapped video that matches the desired target duration while maximally preserving natural video dynamics. Our approach trains a neural network through self-supervision to recognize and accurately localize temporally varying changes in the video playback speed. To re-time videos, we 1. use the model to infer the slowness of individual video frames, and 2. optimize the temporal frame sub-sampling to be consistent with the model's slowness predictions. We demonstrate that this model can detect playback speed variations more accurately while also being orders of magnitude more efficient than prior approaches. Furthermore, we propose an optimization for video re-timing that enables precise control over the target duration and performs more robustly on longer videos than prior methods. We evaluate the model quantitatively on artificially speed-up videos, through transfer to action recognition, and qualitatively through user studies.