Localizing Unseen Activities in Video via Image Query
This work addresses a limitation in video understanding by enabling localization of activities not seen during training, which is incremental as it builds on existing action localization methods.
The paper tackles the problem of localizing unseen activities in untrimmed videos using image queries, proposing a self-attention interaction localizer that addresses challenges like eliminating inessential image content and fuzzy localization, with experiments demonstrating its effectiveness on a new dataset derived from ActivityNet.
Action localization in untrimmed videos is an important topic in the field of video understanding. However, existing action localization methods are restricted to a pre-defined set of actions and cannot localize unseen activities. Thus, we consider a new task to localize unseen activities in videos via image queries, named Image-Based Activity Localization. This task faces three inherent challenges: (1) how to eliminate the influence of semantically inessential contents in image queries; (2) how to deal with the fuzzy localization of inaccurate image queries; (3) how to determine the precise boundaries of target segments. We then propose a novel self-attention interaction localizer to retrieve unseen activities in an end-to-end fashion. Specifically, we first devise a region self-attention method with relative position encoding to learn fine-grained image region representations. Then, we employ a local transformer encoder to build multi-step fusion and reasoning of image and video contents. We next adopt an order-sensitive localizer to directly retrieve the target segment. Furthermore, we construct a new dataset ActivityIBAL by reorganizing the ActivityNet dataset. The extensive experiments show the effectiveness of our method.