Learning Frequency and Memory-Aware Prompts for Multi-Modal Object Tracking
This work addresses the challenge of enhancing cross-modal interaction and temporal coherence in multi-modal tracking for applications like surveillance and autonomous systems, representing an incremental improvement over existing prompt-learning methods.
The paper tackles the problem of underutilizing modality-specific frequency structure and long-range temporal dependencies in multi-modal object tracking by proposing a dual-adapter framework with frequency-guided and memory-aware prompts, achieving state-of-the-art results on RGB-Thermal, RGB-Depth, and RGB-Event benchmarks with improved parameter efficiency and runtime.
Prompt-learning-based multi-modal trackers have made strong progress by using lightweight visual adapters to inject auxiliary-modality cues into frozen foundation models. However, they still underutilize two essentials: modality-specific frequency structure and long-range temporal dependencies. We present Learning Frequency and Memory-Aware Prompts, a dual-adapter framework that injects lightweight prompts into a frozen RGB tracker. A frequency-guided visual adapter adaptively transfers complementary cues across modalities by jointly calibrating spatial, channel, and frequency components, narrowing the modality gap without full fine-tuning. A multilevel memory adapter with short, long, and permanent memory stores, updates, and retrieves reliable temporal context, enabling consistent propagation across frames and robust recovery from occlusion, motion blur, and illumination changes. This unified design preserves the efficiency of prompt learning while strengthening cross-modal interaction and temporal coherence. Extensive experiments on RGB-Thermal, RGB-Depth, and RGB-Event benchmarks show consistent state-of-the-art results over fully fine-tuned and adapter-based baselines, together with favorable parameter efficiency and runtime. Code and models are available at https://github.com/xuboyue1999/mmtrack.git.