Enhancing Transformer Backbone for Egocentric Video Action Segmentation
This work addresses a crucial task for applications in mixed reality, human behavior analysis, and robotics, but it is incremental as it builds on existing transformer-based methods.
The paper tackled the problem of improving transformer backbones for egocentric video action segmentation by introducing a dual dilated attention mechanism and cross-connections between encoder and decoder blocks, resulting in state-of-the-art performance on GTEA and HOI4D Office Tools datasets.
Egocentric temporal action segmentation in videos is a crucial task in computer vision with applications in various fields such as mixed reality, human behavior analysis, and robotics. Although recent research has utilized advanced visual-language frameworks, transformers remain the backbone of action segmentation models. Therefore, it is necessary to improve transformers to enhance the robustness of action segmentation models. In this work, we propose two novel ideas to enhance the state-of-the-art transformer for action segmentation. First, we introduce a dual dilated attention mechanism to adaptively capture hierarchical representations in both local-to-global and global-to-local contexts. Second, we incorporate cross-connections between the encoder and decoder blocks to prevent the loss of local context by the decoder. We also utilize state-of-the-art visual-language representation learning techniques to extract richer and more compact features for our transformer. Our proposed approach outperforms other state-of-the-art methods on the Georgia Tech Egocentric Activities (GTEA) and HOI4D Office Tools datasets, and we validate our introduced components with ablation studies. The source code and supplementary materials are publicly available on https://www.sail-nu.com/dxformer.