CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Models
This work addresses the challenge of costly expert data and limited generalization in EVT for embodied intelligence, offering a novel benchmark and framework for improved robustness, though it is incremental in advancing multi-agent approaches.
The paper tackles the problem of Embodied Visual Tracking (EVT) by proposing CoMaTrack, a competitive multi-agent reinforcement learning framework that trains agents in adversarial settings, resulting in state-of-the-art performance with a 3B VLM surpassing previous 7B model methods on benchmarks like EVT-Bench, achieving scores such as 92.1% in STT.
Embodied Visual Tracking (EVT), a core dynamic task in embodied intelligence, requires an agent to precisely follow a language-specified target. Yet most existing methods rely on single-agent imitation learning, suffering from costly expert data and limited generalization due to static training environments. Inspired by competition-driven capability evolution, we propose CoMaTrack, a competitive game-theoretic multi-agent reinforcement learning framework that trains agents in a dynamic adversarial setting with competitive subtasks, yielding stronger adaptive planning and interference-resilient strategies. We further introduce CoMaTrack-Bench, the first benchmark for competitive EVT, featuring game scenarios between a tracker and adaptive opponents across diverse environments and instructions, enabling standardized robustness evaluation under active adversarial interactions. Experiments show that CoMaTrack achieves state-of-the-art results on both standard benchmarks and CoMaTrack-Bench. Notably, a 3B VLM trained with our framework surpasses previous single-agent imitation learning methods based on 7B models on the challenging EVT-Bench, achieving 92.1% in STT, 74.2% in DT, and 57.5% in AT. The benchmark code will be available at https://github.com/wlqcode/CoMaTrack-Bench