UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity
This addresses the limitation of ambiguous and manual granularity control in vision foundation models like SAM, enabling more precise segmentation for users in computer vision tasks, though it builds incrementally on the UnSAM strategy.
The paper tackles the problem of controlling segmentation granularity in the Segment Anything Model (SAM) family, which typically requires manual refinement, by introducing UnSAMv2, a self-supervised method that enables segment anything at any granularity without human annotations. The result shows substantial improvements, such as reducing NoC90 from 5.69 to 4.75 and increasing 1-IoU from 58.0 to 73.1 across over 11 benchmarks.
The Segment Anything Model (SAM) family has become a widely adopted vision foundation model, but its ability to control segmentation granularity remains limited. Users often need to refine results manually - by adding more prompts or selecting from pre-generated masks - to achieve the desired level of detail. This process can be ambiguous, as the same prompt may correspond to several plausible masks, and collecting dense annotations across all granularities is prohibitively expensive, making supervised solutions infeasible. To address this limitation, we introduce UnSAMv2, which enables segment anything at any granularity without human annotations. UnSAMv2 extends the divide-and-conquer strategy of UnSAM by discovering abundant mask-granularity pairs and introducing a novel granularity control embedding that enables precise, continuous control over segmentation scale. Remarkably, with only $6$K unlabeled images and $0.02\%$ additional parameters, UnSAMv2 substantially enhances SAM-2, achieving segment anything at any granularity across interactive, whole-image, and video segmentation tasks. Evaluated on over $11$ benchmarks, UnSAMv2 improves $\text{NoC}_{90}$ (5.69 $\rightarrow$ 4.75), 1-IoU (58.0 $\rightarrow$ 73.1), and $\text{AR}_{1000}$ (49.6 $\rightarrow$ 68.3), showing that small amounts of unlabeled data with a granularity-aware self-supervised learning method can unlock the potential of vision foundation models.