OW-VISCapTor: Abstractors for Open-World Video Instance Segmentation and Captioning
This addresses the problem of handling novel objects in video analysis for applications like autonomous systems or video understanding, representing a novel task formulation rather than an incremental improvement.
The paper tackles the new task of open-world video instance segmentation and captioning, which involves detecting, segmenting, tracking, and describing unseen objects in videos, and achieves a 13% improvement on unseen objects and a 10% improvement on object-centric captions compared to a baseline.
We propose the new task 'open-world video instance segmentation and captioning'. It requires to detect, segment, track and describe with rich captions never before seen objects. This challenging task can be addressed by developing "abstractors" which connect a vision model and a language foundation model. Concretely, we connect a multi-scale visual feature extractor and a large language model (LLM) by developing an object abstractor and an object-to-text abstractor. The object abstractor, consisting of a prompt encoder and transformer blocks, introduces spatially-diverse open-world object queries to discover never before seen objects in videos. An inter-query contrastive loss further encourages the diversity of object queries. The object-to-text abstractor is augmented with masked cross-attention and acts as a bridge between the object queries and a frozen LLM to generate rich and descriptive object-centric captions for each detected object. Our generalized approach surpasses the baseline that jointly addresses the tasks of open-world video instance segmentation and dense video object captioning by 13% on never before seen objects, and by 10% on object-centric captions.