Fine-Grained Instance-Level Sketch-Based Video Retrieval
This addresses a challenging cross-modal retrieval problem for applications requiring precise matching of visual appearance and motion from sketches to videos, representing an incremental advancement over prior sketch-based or coarse-grained video retrieval methods.
The paper tackles the problem of fine-grained instance-level sketch-based video retrieval, where a sketch sequence is used to retrieve a specific video instance, and introduces a novel multi-stream multi-modality deep network with a relation module that significantly outperforms existing state-of-the-art models.
Existing sketch-analysis work studies sketches depicting static objects or scenes. In this work, we propose a novel cross-modal retrieval problem of fine-grained instance-level sketch-based video retrieval (FG-SBVR), where a sketch sequence is used as a query to retrieve a specific target video instance. Compared with sketch-based still image retrieval, and coarse-grained category-level video retrieval, this is more challenging as both visual appearance and motion need to be simultaneously matched at a fine-grained level. We contribute the first FG-SBVR dataset with rich annotations. We then introduce a novel multi-stream multi-modality deep network to perform FG-SBVR under both strong and weakly supervised settings. The key component of the network is a relation module, designed to prevent model over-fitting given scarce training data. We show that this model significantly outperforms a number of existing state-of-the-art models designed for video analysis.