Do self-supervised speech and language models extract similar representations as human brain?
This research provides insights into the neural basis of speech and language processing and the convergence of self-supervised learning models, but it is incremental as it builds on existing knowledge of brain-model alignment.
The study investigated whether self-supervised speech and language models (Wav2Vec2.0 and GPT-2) extract similar representations as the human brain during speech perception, finding that both models accurately predict brain activity in the auditory cortex with a significant correlation between their predictions, and shared speech contextual information explained most of the variance in brain activity.
Speech and language models trained through self-supervised learning (SSL) demonstrate strong alignment with brain activity during speech and language perception. However, given their distinct training modalities, it remains unclear whether they correlate with the same neural aspects. We directly address this question by evaluating the brain prediction performance of two representative SSL models, Wav2Vec2.0 and GPT-2, designed for speech and language tasks. Our findings reveal that both models accurately predict speech responses in the auditory cortex, with a significant correlation between their brain predictions. Notably, shared speech contextual information between Wav2Vec2.0 and GPT-2 accounts for the majority of explained variance in brain activity, surpassing static semantic and lower-level acoustic-phonetic information. These results underscore the convergence of speech contextual representations in SSL models and their alignment with the neural network underlying speech perception, offering valuable insights into both SSL models and the neural basis of speech and language processing.