UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language
This work addresses the challenge of building versatile brain-computer interfaces by enabling coherent text decoding from cognitive signals, though it is incremental as it builds on existing methods with a unified structure.
The paper tackles the problem of decoding text from cognitive signals like fMRI and EEG, proposing the first open-vocabulary fMRI2text task and a baseline model UniCoRN that achieves a 34.77% BLEU score on fMRI2text and 37.04% on EEG-to-text, surpassing prior baselines.
Decoding text stimuli from cognitive signals (e.g. fMRI) enhances our understanding of the human language system, paving the way for building versatile Brain-Computer Interface. However, existing studies largely focus on decoding individual word-level fMRI volumes from a restricted vocabulary, which is far too idealized for real-world application. In this paper, we propose fMRI2text, the first openvocabulary task aiming to bridge fMRI time series and human language. Furthermore, to explore the potential of this new task, we present a baseline solution, UniCoRN: the Unified Cognitive Signal ReconstructioN for Brain Decoding. By reconstructing both individual time points and time series, UniCoRN establishes a robust encoder for cognitive signals (fMRI & EEG). Leveraging a pre-trained language model as decoder, UniCoRN proves its efficacy in decoding coherent text from fMRI series across various split settings. Our model achieves a 34.77% BLEU score on fMRI2text, and a 37.04% BLEU when generalized to EEGto-text decoding, thereby surpassing the former baseline. Experimental results indicate the feasibility of decoding consecutive fMRI volumes, and the effectiveness of decoding different cognitive signals using a unified structure.