On Moving Object Segmentation from Monocular Video with Transformers
This addresses the challenging problem of segmenting moving objects from single moving cameras for applications like autonomous driving or video analysis, representing an incremental advance with a new method for a known bottleneck.
The paper tackles moving object segmentation from monocular video by proposing M3Former, a novel fusion architecture using transformers, and systematically analyzes motion representations and training data, achieving state-of-the-art performance on Kitti and Davis datasets.
Moving object detection and segmentation from a single moving camera is a challenging task, requiring an understanding of recognition, motion and 3D geometry. Combining both recognition and reconstruction boils down to a fusion problem, where appearance and motion features need to be combined for classification and segmentation. In this paper, we present a novel fusion architecture for monocular motion segmentation - M3Former, which leverages the strong performance of transformers for segmentation and multi-modal fusion. As reconstructing motion from monocular video is ill-posed, we systematically analyze different 2D and 3D motion representations for this problem and their importance for segmentation performance. Finally, we analyze the effect of training data and show that diverse datasets are required to achieve SotA performance on Kitti and Davis.