SFusion: Self-attention based N-to-One Multimodal Fusion Block
This addresses a practical challenge in multimodal AI for applications like healthcare and activity recognition, though it is incremental as it builds on existing self-attention and fusion methods.
The paper tackles the N-to-One multimodal fusion problem where missing modalities vary across situations, proposing SFusion, a self-attention based block that automatically fuses available modalities without synthesizing missing ones, achieving better performance than competing strategies in human activity recognition and brain tumor segmentation tasks.
People perceive the world with different senses, such as sight, hearing, smell, and touch. Processing and fusing information from multiple modalities enables Artificial Intelligence to understand the world around us more easily. However, when there are missing modalities, the number of available modalities is different in diverse situations, which leads to an N-to-One fusion problem. To solve this problem, we propose a self-attention based fusion block called SFusion. Different from preset formulations or convolution based methods, the proposed block automatically learns to fuse available modalities without synthesizing or zero-padding missing ones. Specifically, the feature representations extracted from upstream processing model are projected as tokens and fed into self-attention module to generate latent multimodal correlations. Then, a modal attention mechanism is introduced to build a shared representation, which can be applied by the downstream decision model. The proposed SFusion can be easily integrated into existing multimodal analysis networks. In this work, we apply SFusion to different backbone networks for human activity recognition and brain tumor segmentation tasks. Extensive experimental results show that the SFusion block achieves better performance than the competing fusion strategies. Our code is available at https://github.com/scut-cszcl/SFusion.