Instruction-Driven Fusion of Infrared-Visible Images: Tailoring for Diverse Downstream Tasks
This work addresses the problem of training complexity and performance trade-offs in multi-task image fusion for applications in computer vision, though it appears incremental as it builds on existing fusion methods with adaptive mechanisms.
The paper tackles the challenge of infrared-visible image fusion for multiple downstream tasks by proposing Task-Oriented Adaptive Regulation (T-OAR) and Task-related Dynamic Prompt Injection (T-DPI), which generate task-specific fusion images without separate training, reducing computational costs and enhancing performance in tasks like object detection, semantic segmentation, and salient object detection.
The primary value of infrared and visible image fusion technology lies in applying the fusion results to downstream tasks. However, existing methods face challenges such as increased training complexity and significantly compromised performance of individual tasks when addressing multiple downstream tasks simultaneously. To tackle this, we propose Task-Oriented Adaptive Regulation (T-OAR), an adaptive mechanism specifically designed for multi-task environments. Additionally, we introduce the Task-related Dynamic Prompt Injection (T-DPI) module, which generates task-specific dynamic prompts from user-input text instructions and integrates them into target representations. This guides the feature extraction module to produce representations that are more closely aligned with the specific requirements of downstream tasks. By incorporating the T-DPI module into the T-OAR framework, our approach generates fusion images tailored to task-specific requirements without the need for separate training or task-specific weights. This not only reduces computational costs but also enhances adaptability and performance across multiple tasks. Experimental results show that our method excels in object detection, semantic segmentation, and salient object detection, demonstrating its strong adaptability, flexibility, and task specificity. This provides an efficient solution for image fusion in multi-task environments, highlighting the technology's potential across diverse applications.