Rethinking Cross-Subject Data Splitting for Brain-to-Text Decoding
This addresses a critical methodological flaw in brain-to-text decoding research, which is incremental but essential for ensuring reliable results in neuroscience and AI applications.
The paper identified that current cross-subject data splitting methods in brain-to-text decoding suffer from data leakage, leading to overfitting and overestimated model performance, and proposed a corrected splitting criterion to re-evaluate state-of-the-art models.
Recent major milestones have successfully reconstructed natural language from non-invasive brain signals (e.g. functional Magnetic Resonance Imaging (fMRI) and Electroencephalogram (EEG)) across subjects. However, we find current dataset splitting strategies for cross-subject brain-to-text decoding are wrong. Specifically, we first demonstrate that all current splitting methods suffer from data leakage problem, which refers to the leakage of validation and test data into training set, resulting in significant overfitting and overestimation of decoding models. In this study, we develop a right cross-subject data splitting criterion without data leakage for decoding fMRI and EEG signal to text. Some SOTA brain-to-text decoding models are re-evaluated correctly with the proposed criterion for further research.