Weakly-Supervised Online Action Segmentation in Multi-View Instructional Videos
This work addresses the problem of segmenting actions in streaming instructional videos with weak supervision, which is incremental by building on existing methods for multi-view datasets.
The paper tackles weakly-supervised online action segmentation in instructional videos by introducing a framework with Dynamic Programming and an Online-Offline Discrepancy Loss, achieving improved temporal consistency and exploiting multi-view correspondence for training without extra annotations, with results on Breakfast and IKEA ASM datasets showing efficacy in cooking and assembly domains.
This paper addresses a new problem of weakly-supervised online action segmentation in instructional videos. We present a framework to segment streaming videos online at test time using Dynamic Programming and show its advantages over greedy sliding window approach. We improve our framework by introducing the Online-Offline Discrepancy Loss (OODL) to encourage the segmentation results to have a higher temporal consistency. Furthermore, only during training, we exploit frame-wise correspondence between multiple views as supervision for training weakly-labeled instructional videos. In particular, we investigate three different multi-view inference techniques to generate more accurate frame-wise pseudo ground-truth with no additional annotation cost. We present results and ablation studies on two benchmark multi-view datasets, Breakfast and IKEA ASM. Experimental results show efficacy of the proposed methods both qualitatively and quantitatively in two domains of cooking and assembly.