Optimizing fMRI Data Acquisition for Decoding Natural Speech with Limited Participants
This work addresses challenges in brain-computer interfaces for speech decoding with small datasets, but it is incremental as it builds on existing methods and datasets.
The study tackled the problem of decoding natural speech from fMRI data with limited participants, finding that multi-subject training did not improve accuracy over single-subject approaches, and decoders performed better on syntactic than semantic features.
We investigate optimal strategies for decoding perceived natural speech from fMRI data acquired from a limited number of participants. Leveraging Lebel et al. (2023)'s dataset of 8 participants, we first demonstrate the effectiveness of training deep neural networks to predict LLM-derived text representations from fMRI activity. Then, in this data regime, we observe that multi-subject training does not improve decoding accuracy compared to single-subject approach. Furthermore, training on similar or different stimuli across subjects has a negligible effect on decoding accuracy. Finally, we find that our decoders better model syntactic than semantic features, and that stories containing sentences with complex syntax or rich semantic content are more challenging to decode. While our results demonstrate the benefits of having extensive data per participant (deep phenotyping), they suggest that leveraging multi-subject for natural speech decoding likely requires deeper phenotyping or a substantially larger cohort.