SDI-Paste: Synthetic Dynamic Instance Copy-Paste for Video Instance Segmentation
This addresses the problem of limited training data for video instance segmentation researchers, offering an incremental improvement through synthetic augmentation.
The paper tackles the lack of scalable Copy-Paste data augmentation for video tasks by developing a pipeline that uses generative models to create and insert synthetic dynamic objects into videos, achieving gains of +2.9 AP (6.5%) and +2.1 AP (4.9%) on the Youtube-VIS 2021 dataset for Video Instance Segmentation.
Data augmentation methods such as Copy-Paste have been studied as effective ways to expand training datasets while incurring minimal costs. While such methods have been extensively implemented for image level tasks, we found no scalable implementation of Copy-Paste built specifically for video tasks. In this paper, we leverage the recent growth in video fidelity of generative models to explore effective ways of incorporating synthetically generated objects into existing video datasets to artificially expand object instance pools. We first procure synthetic video sequences featuring objects that morph dynamically with time. Our carefully devised pipeline automatically segments then copy-pastes these dynamic instances across the frames of any target background video sequence. We name our video data augmentation pipeline Synthetic Dynamic Instance Copy-Paste, and test it on the complex task of Video Instance Segmentation which combines detection, segmentation and tracking of object instances across a video sequence. Extensive experiments on the popular Youtube-VIS 2021 dataset using two separate popular networks as baselines achieve strong gains of +2.9 AP (6.5%) and +2.1 AP (4.9%). We make our code and models publicly available.