There is no SAMantics! Exploring SAM as a Backbone for Visual Understanding Tasks
This work addresses the limitation of SAM for researchers and practitioners in computer vision who need semantic understanding in tasks like classification.
The study investigated whether the Segment Anything Model (SAM) has inherent semantic understanding for visual tasks, finding that SAM's features lack semantic discriminability, limiting class differentiation, and proposed a training-free method using DINOv2 features to improve it.
The Segment Anything Model (SAM) was originally designed for label-agnostic mask generation. Does this model also possess inherent semantic understanding, of value to broader visual tasks? In this work we follow a multi-staged approach towards exploring this question. We firstly quantify SAM's semantic capabilities by comparing base image encoder efficacy under classification tasks, in comparison with established models (CLIP and DINOv2). Our findings reveal a significant lack of semantic discriminability in SAM feature representations, limiting potential for tasks that require class differentiation. This initial result motivates our exploratory study that attempts to enable semantic information via in-context learning with lightweight fine-tuning where we observe that generalisability to unseen classes remains limited. Our observations culminate in the proposal of a training-free approach that leverages DINOv2 features, towards better endowing SAM with semantic understanding and achieving instance-level class differentiation through feature-based similarity. Our study suggests that incorporation of external semantic sources provides a promising direction for the enhancement of SAM's utility with respect to complex visual tasks that require semantic understanding.