DC-ViT: Modulating Spatial and Channel Interactions for Multi-Channel Images
This addresses the challenge of preserving channel-specific semantics in multi-channel imaging for applications like medical or remote sensing, representing an incremental advance over prior MC-ViT approaches.
The paper tackled the problem of feature dilution in multi-channel imaging due to unrestricted token interactions across channels, proposing DC-ViT with Decoupled Self-Attention and Decoupled Aggregation, which achieved consistent improvements over existing methods in experiments across three benchmarks.
Training and evaluation in multi-channel imaging (MCI) remains challenging due to heterogeneous channel configurations arising from varying staining protocols, sensor types, and acquisition settings. This heterogeneity limits the applicability of fixed-channel encoders commonly used in general computer vision. Recent Multi-Channel Vision Transformers (MC-ViTs) address this by enabling flexible channel inputs, typically by jointly encoding patch tokens from all channels within a unified attention space. However, unrestricted token interactions across channels can lead to feature dilution, reducing the ability to preserve channel-specific semantics that are critical in MCI data. To address this, we propose Decoupled Vision Transformer (DC-ViT), which explicitly regulates information sharing using Decoupled Self-Attention (DSA), which decomposes token updates into two complementary pathways: spatial updates that model intra-channel structure, and channel-wise updates that adaptively integrate cross-channel information. This decoupling mitigates informational collapse while allowing selective inter-channel interaction. To further exploit these enhanced channel-specific representations, we introduce Decoupled Aggregation (DAG), which allows the model to learn task-specific channel importances. Extensive experiments across three MCI benchmarks demonstrate consistent improvements over existing MC-ViT approaches.