Leveraging Vision-Language Models for Open-Vocabulary Instance Segmentation and Tracking
This work addresses the need for robust visual understanding in dynamic environments, such as robotics, by combining descriptive and grounding capabilities, though it is incremental as it builds on existing vision-language and segmentation techniques.
The paper tackled the problem of open-vocabulary instance segmentation and tracking by integrating vision-language models with detection and segmentation methods, achieving real-time processing with minimal computational overhead and demonstrating broad applicability across datasets and robotics platforms.
Vision-language models (VLMs) excel in visual understanding but often lack reliable grounding capabilities and actionable inference rates. Integrating them with open-vocabulary object detection (OVD), instance segmentation, and tracking leverages their strengths while mitigating these drawbacks. We utilize VLM-generated structured descriptions to identify visible object instances, collect application-relevant attributes, and inform an open-vocabulary detector to extract corresponding bounding boxes that are passed to a video segmentation model providing segmentation masks and tracking. Once initialized, this model directly extracts segmentation masks, processing image streams in real time with minimal computational overhead. Tracks can be updated online as needed by generating new structured descriptions and detections. This combines the descriptive power of VLMs with the grounding capability of OVD and the pixel-level understanding and speed of video segmentation. Our evaluation across datasets and robotics platforms demonstrates the broad applicability of this approach, showcasing its ability to extract task-specific attributes from non-standard objects in dynamic environments. Code, data, videos, and benchmarks are available at https://vlm-gist.github.io