Where am I? Cross-View Geo-localization with Natural Language Descriptions
This addresses a problem for applications like pedestrian navigation and emergency response by enabling text-guided geo-localization, though it is incremental as it builds on existing cross-view methods.
The paper tackles cross-view geo-localization by introducing a task that uses natural language descriptions to retrieve satellite images or OSM data, constructing the CVG-Text dataset and proposing the CrossText2Loc method, which improves recall by 10% and offers explainable retrieval reasons.
Cross-view geo-localization identifies the locations of street-view images by matching them with geo-tagged satellite images or OSM. However, most existing studies focus on image-to-image retrieval, with fewer addressing text-guided retrieval, a task vital for applications like pedestrian navigation and emergency response. In this work, we introduce a novel task for cross-view geo-localization with natural language descriptions, which aims to retrieve corresponding satellite images or OSM database based on scene text descriptions. To support this task, we construct the CVG-Text dataset by collecting cross-view data from multiple cities and employing a scene text generation approach that leverages the annotation capabilities of Large Multimodal Models to produce high-quality scene text descriptions with localization details. Additionally, we propose a novel text-based retrieval localization method, CrossText2Loc, which improves recall by 10% and demonstrates excellent long-text retrieval capabilities. In terms of explainability, it not only provides similarity scores but also offers retrieval reasons. More information can be found at https://yejy53.github.io/CVG-Text/ .