MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
This addresses the problem of accurately locating and classifying actions in long videos for applications like surveillance and robotics, representing an incremental improvement over existing temporal convolution methods.
The paper tackles action segmentation in untrimmed videos by introducing a multi-stage temporal convolutional network (MS-TCN) that refines predictions across stages, achieving state-of-the-art results on datasets like 50Salads, GTEA, and Breakfast.
Temporally locating and classifying action segments in long untrimmed videos is of particular interest to many applications like surveillance and robotics. While traditional approaches follow a two-step pipeline, by generating frame-wise probabilities and then feeding them to high-level temporal models, recent approaches use temporal convolutions to directly classify the video frames. In this paper, we introduce a multi-stage architecture for the temporal action segmentation task. Each stage features a set of dilated temporal convolutions to generate an initial prediction that is refined by the next one. This architecture is trained using a combination of a classification loss and a proposed smoothing loss that penalizes over-segmentation errors. Extensive evaluation shows the effectiveness of the proposed model in capturing long-range dependencies and recognizing action segments. Our model achieves state-of-the-art results on three challenging datasets: 50Salads, Georgia Tech Egocentric Activities (GTEA), and the Breakfast dataset.