Wills Aligner: Multi-Subject Collaborative Brain Visual Decoding
This addresses the challenge of personalization in brain visual decoding, enabling broader applicability in real-world scenarios for neuroscience and brain-computer interface applications, though it appears incremental as it builds on existing deep learning models.
The paper tackles the problem of decoding visual information from human brain activity across diverse individuals by introducing Wills Aligner, a multi-subject collaborative approach that aligns fMRI data and uses mixture-of-brain-expert adapters with meta-learning, achieving promising performance in tasks like classification, cross-modal retrieval, and image reconstruction.
Decoding visual information from human brain activity has seen remarkable advancements in recent research. However, the diversity in cortical parcellation and fMRI patterns across individuals has prompted the development of deep learning models tailored to each subject. The personalization limits the broader applicability of brain visual decoding in real-world scenarios. To address this issue, we introduce Wills Aligner, a novel approach designed to achieve multi-subject collaborative brain visual decoding. Wills Aligner begins by aligning the fMRI data from different subjects at the anatomical level. It then employs delicate mixture-of-brain-expert adapters and a meta-learning strategy to account for individual fMRI pattern differences. Additionally, Wills Aligner leverages the semantic relation of visual stimuli to guide the learning of inter-subject commonality, enabling visual decoding for each subject to draw insights from other subjects' data. We rigorously evaluate our Wills Aligner across various visual decoding tasks, including classification, cross-modal retrieval, and image reconstruction. The experimental results demonstrate that Wills Aligner achieves promising performance.