Spatio-temporal Person Retrieval via Natural Language Queries
This addresses the problem of locating people in videos based on text descriptions for applications like surveillance or video search, but it is incremental as it builds on existing detection and retrieval techniques.
The paper tackles spatio-temporal person retrieval from videos using natural language queries by outputting bounding box tubes, introducing a new dataset with video annotations and descriptions. It presents a model combining detection and retrieval methods, achieving strong baseline results and demonstrating versatility in other tasks.
In this paper, we address the problem of spatio-temporal person retrieval from multiple videos using a natural language query, in which we output a tube (i.e., a sequence of bounding boxes) which encloses the person described by the query. For this problem, we introduce a novel dataset consisting of videos containing people annotated with bounding boxes for each second and with five natural language descriptions. To retrieve the tube of the person described by a given natural language query, we design a model that combines methods for spatio-temporal human detection and multimodal retrieval. We conduct comprehensive experiments to compare a variety of tube and text representations and multimodal retrieval methods, and present a strong baseline in this task as well as demonstrate the efficacy of our tube representation and multimodal feature embedding technique. Finally, we demonstrate the versatility of our model by applying it to two other important tasks.