Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptions
This addresses the problem of creating enriched spatiotemporal descriptions for videos, which is incremental as it builds on existing video captioning methods.
The paper tackles the challenge of generating detailed video captions by introducing Video ChatCaptioner, which uses ChatGPT to select frames and ask content-driven questions, resulting in captions with more visual details as demonstrated qualitatively.
Video captioning aims to convey dynamic scenes from videos using natural language, facilitating the understanding of spatiotemporal information within our environment. Although there have been recent advances, generating detailed and enriched video descriptions continues to be a substantial challenge. In this work, we introduce Video ChatCaptioner, an innovative approach for creating more comprehensive spatiotemporal video descriptions. Our method employs a ChatGPT model as a controller, specifically designed to select frames for posing video content-driven questions. Subsequently, a robust algorithm is utilized to answer these visual queries. This question-answer framework effectively uncovers intricate video details and shows promise as a method for enhancing video content. Following multiple conversational rounds, ChatGPT can summarize enriched video content based on previous conversations. We qualitatively demonstrate that our Video ChatCaptioner can generate captions containing more visual details about the videos. The code is publicly available at https://github.com/Vision-CAIR/ChatCaptioner