Towards Sequence-Level Training for Visual Tracking
This work addresses a fundamental mismatch in training for visual tracking, which is an incremental improvement for researchers and practitioners in computer vision.
The paper tackles the inconsistency between frame-level training and the sequence-level nature of visual object tracking by introducing a sequence-level training strategy based on reinforcement learning, resulting in consistent accuracy and robustness improvements for four representative tracking models on standard benchmarks like LaSOT, TrackingNet, and GOT-10k.
Despite the extensive adoption of machine learning on the task of visual object tracking, recent learning-based approaches have largely overlooked the fact that visual tracking is a sequence-level task in its nature; they rely heavily on frame-level training, which inevitably induces inconsistency between training and testing in terms of both data distributions and task objectives. This work introduces a sequence-level training strategy for visual tracking based on reinforcement learning and discusses how a sequence-level design of data sampling, learning objectives, and data augmentation can improve the accuracy and robustness of tracking algorithms. Our experiments on standard benchmarks including LaSOT, TrackingNet, and GOT-10k demonstrate that four representative tracking models, SiamRPN++, SiamAttn, TransT, and TrDiMP, consistently improve by incorporating the proposed methods in training without modifying architectures.