Det-SAM2:Technical Report on the Self-Prompting Segmentation Framework Based on Segment Anything Model 2
This work addresses the need for higher automation in practical video segmentation applications, such as AI refereeing in billiards, but it is incremental as it builds directly on the existing SAM2 framework.
The authors tackled the challenge of automating video segmentation by developing Det-SAM2, a pipeline that uses a detection model to generate prompts for SAM2, enabling inference on infinitely long video streams with constant memory usage while maintaining the efficiency and accuracy of SAM2.
Segment Anything Model 2 (SAM2) demonstrates exceptional performance in video segmentation and refinement of segmentation results. We anticipate that it can further evolve to achieve higher levels of automation for practical applications. Building upon SAM2, we conducted a series of practices that ultimately led to the development of a fully automated pipeline, termed Det-SAM2, in which object prompts are automatically generated by a detection model to facilitate inference and refinement by SAM2. This pipeline enables inference on infinitely long video streams with constant VRAM and RAM usage, all while preserving the same efficiency and accuracy as the original SAM2. This technical report focuses on the construction of the overall Det-SAM2 framework and the subsequent engineering optimization applied to SAM2. We present a case demonstrating an application built on the Det-SAM2 framework: AI refereeing in a billiards scenario, derived from our business context. The project at \url{https://github.com/motern88/Det-SAM2}.