Symmetric Network with Spatial Relationship Modeling for Natural Language-based Vehicle Retrieval
This work addresses the problem of retrieving vehicles using text descriptions for applications like surveillance and traffic management, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles natural language-based vehicle retrieval by proposing a symmetric network with spatial relationship modeling to incorporate vehicle appearance, trajectory, and surrounding environment, achieving 43.92% MRR accuracy and first place in the AI City Challenge.
Natural language (NL) based vehicle retrieval aims to search specific vehicle given text description. Different from the image-based vehicle retrieval, NL-based vehicle retrieval requires considering not only vehicle appearance, but also surrounding environment and temporal relations. In this paper, we propose a Symmetric Network with Spatial Relationship Modeling (SSM) method for NL-based vehicle retrieval. Specifically, we design a symmetric network to learn the unified cross-modal representations between text descriptions and vehicle images, where vehicle appearance details and vehicle trajectory global information are preserved. Besides, to make better use of location information, we propose a spatial relationship modeling methods to take surrounding environment and mutual relationship between vehicles into consideration. The qualitative and quantitative experiments verify the effectiveness of the proposed method. We achieve 43.92% MRR accuracy on the test set of the 6th AI City Challenge on natural language-based vehicle retrieval track, yielding the 1st place among all valid submissions on the public leaderboard. The code is available at https://github.com/hbchen121/AICITY2022_Track2_SSM.