Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation
This work addresses the problem of accurately segmenting actions in videos for applications like surveillance or human-computer interaction, presenting a novel unified approach that is incremental in its method adaptations.
The paper tackles video action segmentation by framing it as a sequence-to-sequence translation task, achieving state-of-the-art or competitive performance on multiple datasets in both fully and timestamp supervised settings.
This paper introduces a unified framework for video action segmentation via sequence to sequence (seq2seq) translation in a fully and timestamp supervised setup. In contrast to current state-of-the-art frame-level prediction methods, we view action segmentation as a seq2seq translation task, i.e., mapping a sequence of video frames to a sequence of action segments. Our proposed method involves a series of modifications and auxiliary loss functions on the standard Transformer seq2seq translation model to cope with long input sequences opposed to short output sequences and relatively few videos. We incorporate an auxiliary supervision signal for the encoder via a frame-wise loss and propose a separate alignment decoder for an implicit duration prediction. Finally, we extend our framework to the timestamp supervised setting via our proposed constrained k-medoids algorithm to generate pseudo-segmentations. Our proposed framework performs consistently on both fully and timestamp supervised settings, outperforming or competing state-of-the-art on several datasets. Our code is publicly available at https://github.com/boschresearch/UVAST.