A Multimodal Visual Encoding Model Aided by Introducing Verbal Semantic Information
This work addresses a gap in computational neuroscience by better simulating brain visual processing, though it is incremental as it builds on existing models by adding multimodal features.
The paper tackles the problem of visual encoding models lacking verbal semantic information by proposing a multimodal network that integrates stimulus images with associated textual information, resulting in performance improvements of approximately 15.87% in the left hemisphere and 4.6% in the right hemisphere for voxel prediction.
Biological research has revealed that the verbal semantic information in the brain cortex, as an additional source, participates in nonverbal semantic tasks, such as visual encoding. However, previous visual encoding models did not incorporate verbal semantic information, contradicting this biological finding. This paper proposes a multimodal visual information encoding network model based on stimulus images and associated textual information in response to this issue. Our visual information encoding network model takes stimulus images as input and leverages textual information generated by a text-image generation model as verbal semantic information. This approach injects new information into the visual encoding model. Subsequently, a Transformer network aligns image and text feature information, creating a multimodal feature space. A convolutional network then maps from this multimodal feature space to voxel space, constructing the multimodal visual information encoding network model. Experimental results demonstrate that the proposed multimodal visual information encoding network model outperforms previous models under the exact training cost. In voxel prediction of the left hemisphere of subject 1's brain, the performance improves by approximately 15.87%, while in the right hemisphere, the performance improves by about 4.6%. The multimodal visual encoding network model exhibits superior encoding performance. Additionally, ablation experiments indicate that our proposed model better simulates the brain's visual information processing.