Duala: Dual-Level Alignment of Subjects and Stimuli for Cross-Subject fMRI Decoding
This work provides a method to improve the scalability and practicality of brain-computer interfaces by enabling more robust cross-subject visual decoding from fMRI, particularly for new subjects with limited data.
This paper tackles the problem of cross-subject visual decoding from fMRI data, which typically suffers from degraded performance when adapting to new subjects with limited data. The proposed Duala framework achieves over 81.1% image-to-brain retrieval accuracy on the Natural Scenes Dataset (NSD) with only one hour of fMRI data, outperforming existing fine-tuning strategies.
Cross-subject visual decoding aims to reconstruct visual experiences from brain activity across individuals, enabling more scalable and practical brain-computer interfaces. However, existing methods often suffer from degraded performance when adapting to new subjects with limited data, as they struggle to preserve both the semantic consistency of stimuli and the alignment of brain responses. To address these challenges, we propose Duala, a dual-level alignment framework designed to achieve stimulus-level consistency and subject-level alignment in fMRI-based cross-subject visual decoding. (1) At the stimulus level, Duala introduces a semantic alignment and relational consistency strategy that preserves intra-class similarity and inter-class separability, maintaining clear semantic boundaries during adaptation. (2) At the subject level, a distribution-based feature perturbation mechanism is developed to capture both global and subject-specific variations, enabling adaptation to individual neural representations without overfitting. Experiments on the Natural Scenes Dataset (NSD) demonstrate that Duala effectively improves alignment across subjects. Remarkably, even when fine-tuned with only about one hour of fMRI data, Duala achieves over 81.1% image-to-brain retrieval accuracy and consistently outperforms existing fine-tuning strategies in both retrieval and reconstruction. Our code is available at https://github.com/ShumengLI/Duala.