Target-Grounded Graph-Aware Transformer for Aerial Vision-and-Dialog Navigation
This addresses drone navigation using dialog history for aerial vision, though it appears incremental as it builds on existing transformer and graph methods for a specific competition.
The authors tackled the Aerial Navigation from Dialog History task by proposing a Target-Grounded Graph-Aware Transformer framework, which won the AVDN Challenge with 2.2% and 3.0% absolute improvements in SPL and SR metrics over the baseline.
This report details the methods of the winning entry of the AVDN Challenge in ICCV CLVL 2023. The competition addresses the Aerial Navigation from Dialog History (ANDH) task, which requires a drone agent to associate dialog history with aerial observations to reach the destination. For better cross-modal grounding abilities of the drone agent, we propose a Target-Grounded Graph-Aware Transformer (TG-GAT) framework. Concretely, TG-GAT first leverages a graph-aware transformer to capture spatiotemporal dependency, which benefits navigation state tracking and robust action planning. In addition,an auxiliary visual grounding task is devised to boost the agent's awareness of referred landmarks. Moreover, a hybrid augmentation strategy based on large language models is utilized to mitigate data scarcity limitations. Our TG-GAT framework won the AVDN Challenge, with 2.2% and 3.0% absolute improvements over the baseline on SPL and SR metrics, respectively. The code is available at https://github.com/yifeisu/TG-GAT.