Actor and Action Modular Network for Text-based Video Segmentation
This work improves video segmentation for applications like video editing and surveillance by enabling fine-grained alignment of actors and actions from textual queries, though it is incremental as it builds on existing multi-modal fusion methods.
The paper tackles the problem of text-based video segmentation by addressing semantic asymmetry between video and text, proposing an actor and action modular network that separately localizes actor and action to align modalities, achieving state-of-the-art performance on A2D Sentences and J-HMDB Sentences datasets.
Text-based video segmentation aims to segment an actor in video sequences by specifying the actor and its performing action with a textual query. Previous methods fail to explicitly align the video content with the textual query in a fine-grained manner according to the actor and its action, due to the problem of \emph{semantic asymmetry}. The \emph{semantic asymmetry} implies that two modalities contain different amounts of semantic information during the multi-modal fusion process. To alleviate this problem, we propose a novel actor and action modular network that individually localizes the actor and its action in two separate modules. Specifically, we first learn the actor-/action-related content from the video and textual query, and then match them in a symmetrical manner to localize the target tube. The target tube contains the desired actor and action which is then fed into a fully convolutional network to predict segmentation masks of the actor. Our method also establishes the association of objects cross multiple frames with the proposed temporal proposal aggregation mechanism. This enables our method to segment the video effectively and keep the temporal consistency of predictions. The whole model is allowed for joint learning of the actor-action matching and segmentation, as well as achieves the state-of-the-art performance for both single-frame segmentation and full video segmentation on A2D Sentences and J-HMDB Sentences datasets.