Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Prompting
This addresses the limitation of existing video captioning methods that lack object-level precision, enabling more detailed and temporally coherent descriptions for applications in video analysis and understanding.
The authors tackled the problem of generating fine-grained, object-centric captions for user-selected objects in videos by introducing CAT-V, a training-free framework that integrates segmentation, temporal analysis, and captioning components, achieving detailed descriptions of attributes, actions, and interactions without additional training data.
We present CAT-V (Caption AnyThing in Video), a training-free framework for fine-grained object-centric video captioning that enables detailed descriptions of user-selected objects through time. CAT-V integrates three key components: a Segmenter based on SAMURAI for precise object segmentation across frames, a Temporal Analyzer powered by TRACE-Uni for accurate event boundary detection and temporal analysis, and a Captioner using InternVL-2.5 for generating detailed object-centric descriptions. Through spatiotemporal visual prompts and chain-of-thought reasoning, our framework generates detailed, temporally-aware descriptions of objects' attributes, actions, statuses, interactions, and environmental contexts without requiring additional training data. CAT-V supports flexible user interactions through various visual prompts (points, bounding boxes, and irregular regions) and maintains temporal sensitivity by tracking object states and interactions across different time segments. Our approach addresses limitations of existing video captioning methods, which either produce overly abstract descriptions or lack object-level precision, enabling fine-grained, object-specific descriptions while maintaining temporal coherence and spatial accuracy. The GitHub repository for this project is available at https://github.com/yunlong10/CAT-V