SAM Guided Semantic and Motion Changed Region Mining for Remote Sensing Change Captioning
This work addresses the challenge of generating natural language descriptions of changes in remote sensing images, which is important for applications like environmental monitoring, but it is incremental as it builds on existing methods by incorporating SAM and knowledge graphs.
The paper tackled the problem of remote sensing change captioning by using the SAM foundation model to extract region-level representations and a knowledge graph for object information, achieving state-of-the-art performance on multiple benchmark datasets.
Remote sensing change captioning is an emerging and popular research task that aims to describe, in natural language, the content of interest that has changed between two remote sensing images captured at different times. Existing methods typically employ CNNs/Transformers to extract visual representations from the given images or incorporate auxiliary tasks to enhance the final results, with weak region awareness and limited temporal alignment. To address these issues, this paper explores the use of the SAM (Segment Anything Model) foundation model to extract region-level representations and inject region-of-interest knowledge into the captioning framework. Specifically, we employ a CNN/Transformer model to extract global-level vision features, leverage the SAM foundation model to delineate semantic- and motion-level change regions, and utilize a specially constructed knowledge graph to provide information about objects of interest. These heterogeneous sources of information are then fused via cross-attention, and a Transformer decoder is used to generate the final natural language description of the observed changes. Extensive experimental results demonstrate that our method achieves state-of-the-art performance across multiple widely used benchmark datasets. The source code of this paper will be released on https://github.com/Event-AHU/SAM_ChangeCaptioning