Cross-Enhancement Transformer for Action Segmentation
This work improves action segmentation for video analysis, but it is incremental as it builds on existing transformer and convolutional methods.
The paper tackles the problem of action segmentation by addressing the loss of local information in deep temporal convolutions, proposing a Cross-Enhancement Transformer that integrates local and global information to achieve state-of-the-art performance on three datasets.
Temporal convolutions have been the paradigm of choice in action segmentation, which enhances long-term receptive fields by increasing convolution layers. However, high layers cause the loss of local information necessary for frame recognition. To solve the above problem, a novel encoder-decoder structure is proposed in this paper, called Cross-Enhancement Transformer. Our approach can be effective learning of temporal structure representation with interactive self-attention mechanism. Concatenated each layer convolutional feature maps in encoder with a set of features in decoder produced via self-attention. Therefore, local and global information are used in a series of frame actions simultaneously. In addition, a new loss function is proposed to enhance the training process that penalizes over-segmentation errors. Experiments show that our framework performs state-of-the-art on three challenging datasets: 50Salads, Georgia Tech Egocentric Activities and the Breakfast dataset.