Accelerating Video Object Segmentation with Compressed Video
This work addresses efficiency for video analysis applications, but it is incremental as it builds on existing segmentation algorithms.
The paper tackles the problem of slow video object segmentation by exploiting compressed video bitstreams to accelerate mask propagation, achieving up to 3.5x speed-up with minor accuracy drops on benchmarks like DAVIS17 and YouTube-VOS.
We propose an efficient plug-and-play acceleration framework for semi-supervised video object segmentation by exploiting the temporal redundancies in videos presented by the compressed bitstream. Specifically, we propose a motion vector-based warping method for propagating segmentation masks from keyframes to other frames in a bi-directional and multi-hop manner. Additionally, we introduce a residual-based correction module that can fix wrongly propagated segmentation masks from noisy or erroneous motion vectors. Our approach is flexible and can be added on top of several existing video object segmentation algorithms. We achieved highly competitive results on DAVIS17 and YouTube-VOS on various base models with substantial speed-ups of up to 3.5X with minor drops in accuracy.