SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model
This work addresses fragmentation issues in optical flow estimation for computer vision applications, representing an incremental improvement by adapting a pre-existing model.
The paper tackles the problem of fragmented motion estimation in optical flow by integrating the Segment Anything Model (SAM) into FlowFormer to enhance object perception, achieving state-of-the-art performance with 0.86/2.10 clean/final EPE on Sintel and 3.55/12.32 EPE/F1-all on KITTI-15, surpassing FlowFormer by 8.5%/9.9% and 13.2%/16.3%.
Optical Flow Estimation aims to find the 2D dense motion field between two frames. Due to the limitation of model structures and training datasets, existing methods often rely too much on local clues and ignore the integrity of objects, resulting in fragmented motion estimation. Through theoretical analysis, we find the pre-trained large vision models are helpful in optical flow estimation, and we notice that the recently famous Segment Anything Model (SAM) demonstrates a strong ability to segment complete objects, which is suitable for solving the fragmentation problem. We thus propose a solution to embed the frozen SAM image encoder into FlowFormer to enhance object perception. To address the challenge of in-depth utilizing SAM in non-segmentation tasks like optical flow estimation, we propose an Optical Flow Task-Specific Adaption scheme, including a Context Fusion Module to fuse the SAM encoder with the optical flow context encoder, and a Context Adaption Module to adapt the SAM features for optical flow task with Learned Task-Specific Embedding. Our proposed SAMFlow model reaches 0.86/2.10 clean/final EPE and 3.55/12.32 EPE/F1-all on Sintel and KITTI-15 training set, surpassing Flowformer by 8.5%/9.9% and 13.2%/16.3%. Furthermore, our model achieves state-of-the-art performance on the Sintel and KITTI-15 benchmarks, ranking #1 among all two-frame methods on Sintel clean pass.