SVIP: Sequence VerIfication for Procedures in Videos
This addresses a challenging open-set verification task for video analysis, with incremental contributions in dataset reorganization and evaluation metrics.
The paper tackles the problem of verifying whether two videos perform the same sequence of actions, distinguishing them from videos with step-level transformations but the same overall task, and introduces a baseline method that outperforms existing action recognition approaches.
In this paper, we propose a novel sequence verification task that aims to distinguish positive video pairs performing the same action sequence from negative ones with step-level transformations but still conducting the same task. Such a challenging task resides in an open-set setting without prior action detection or segmentation that requires event-level or even frame-level annotations. To that end, we carefully reorganize two publicly available action-related datasets with step-procedure-task structure. To fully investigate the effectiveness of any method, we collect a scripted video dataset enumerating all kinds of step-level transformations in chemical experiments. Besides, a novel evaluation metric Weighted Distance Ratio is introduced to ensure equivalence for different step-level transformations during evaluation. In the end, a simple but effective baseline based on the transformer encoder with a novel sequence alignment loss is introduced to better characterize long-term dependency between steps, which outperforms other action recognition methods. Codes and data will be released.