TASeg: Text-aware RGB-T Semantic Segmentation based on Fine-tuning Vision Foundation Models
This work addresses semantic segmentation for intelligent systems in open environments, offering an incremental improvement by integrating text information into RGB-T models.
The paper tackled the problem of inaccurate semantic segmentation in RGB-T (RGB and thermal) images when categories share similar visual characteristics by proposing TASeg, a framework that fine-tunes vision foundation models with text embeddings, achieving superior performance in challenging scenarios with fewer trainable parameters.
Reliable semantic segmentation of open environments is essential for intelligent systems, yet significant problems remain: 1) Existing RGB-T semantic segmentation models mainly rely on low-level visual features and lack high-level textual information, which struggle with accurate segmentation when categories share similar visual characteristics. 2) While SAM excels in instance-level segmentation, integrating it with thermal images and text is hindered by modality heterogeneity and computational inefficiency. To address these, we propose TASeg, a text-aware RGB-T segmentation framework by using Low-Rank Adaptation (LoRA) fine-tuning technology to adapt vision foundation models. Specifically, we propose a Dynamic Feature Fusion Module (DFFM) in the image encoder, which effectively merges features from multiple visual modalities while freezing SAM's original transformer blocks. Additionally, we incorporate CLIP-generated text embeddings in the mask decoder to enable semantic alignment, which further rectifies the classification error and improves the semantic understanding accuracy. Experimental results across diverse datasets demonstrate that our method achieves superior performance in challenging scenarios with fewer trainable parameters.