Not All Patches are Equal: Sampling Matters for Visible-Infrared Pre-Training
For researchers working on multi-sensor perception, this paper introduces a sampling perspective to improve VIS-IR pre-training, offering a plug-and-play method that enhances downstream tasks.
The paper addresses the problem that uniform patch-wise contrastive learning for visible-infrared (VIS-IR) alignment is unreliable due to imaging-physics differences. They propose Importance-Aware Sampling (IAS) which adjusts training emphasis based on patch reliability, achieving consistent improvements over strong baselines on multiple VIS-IR benchmarks including IR semantic segmentation, IR object detection, VIS semantic segmentation, and cross-modal retrieval.
Visible-infrared (VIS-IR) alignment is a key pre-training task for robust multi-sensor perception. Most existing methods use uniform patch-wise contrastive learning, but this can be unreliable in VIS-IR data because imaging-physics differences make some spatially paired regions inherently less comparable, and aligning them with equal strength hinders representation learning and downstream transfer. In this paper, we revisit VIS-IR pre-training from a sampling perspective and propose Importance-Aware Sampling (IAS), which adjusts training emphasis based on patch reliability. Specifically, IAS (i) derives patch weights from infrared structural cues and uses them to reweight the contrastive objective; (ii) learns a soft importance mask with a lightweight sampler, optionally warm-started from the hand-crafted prior; and (iii) employs a patch curriculum learning strategy that gradually expands from high-reliability regions to harder patches. It is worth noting that IAS is plug-and-play and works with both patch-/correlation-level alignment (e.g., UNIV-style) and image-level contrastive baselines (e.g., ImageBind-style). Extensive experiments on multiple VIS-IR benchmarks demonstrate consistent improvements over strong baselines, including for IR semantic segmentation, IR object detection and VIS semantic segmentation and cross-modal retrieval task. Code will be released on https://github.com/KlayMa527/IAS.