Re^2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localization
This work addresses a memory bottleneck in video analysis for researchers and practitioners, offering an incremental improvement by enabling efficient end-to-end training with existing models.
The paper tackles the challenge of training temporal action localization (TAL) models end-to-end on long videos due to GPU memory constraints, proposing Re^2TAL, a method that rewires pretrained video backbones with reversible modules to clear intermediate activations from memory, achieving state-of-the-art results of 37.01% average mAP on ActivityNet-v1.3 and 64.9% mAP at tIoU=0.5 on THUMOS-14.
Temporal action localization (TAL) requires long-form reasoning to predict actions of various durations and complex content. Given limited GPU memory, training TAL end to end (i.e., from videos to predictions) on long videos is a significant challenge. Most methods can only train on pre-extracted features without optimizing them for the localization problem, consequently limiting localization performance. In this work, to extend the potential in TAL networks, we propose a novel end-to-end method Re2TAL, which rewires pretrained video backbones for reversible TAL. Re2TAL builds a backbone with reversible modules, where the input can be recovered from the output such that the bulky intermediate activations can be cleared from memory during training. Instead of designing one single type of reversible module, we propose a network rewiring mechanism, to transform any module with a residual connection to a reversible module without changing any parameters. This provides two benefits: (1) a large variety of reversible networks are easily obtained from existing and even future model designs, and (2) the reversible models require much less training effort as they reuse the pre-trained parameters of their original non-reversible versions. Re2TAL, only using the RGB modality, reaches 37.01% average mAP on ActivityNet-v1.3, a new state-of-the-art record, and mAP 64.9% at tIoU=0.5 on THUMOS-14, outperforming all other RGB-only methods.