Perception Activator: An intuitive and portable framework for brain cognitive exploration
This work addresses the challenge of better understanding brain visual perception patterns for neuroscience and AI researchers, but it is incremental as it builds on existing decoding methods by adding fMRI interventions.
The authors tackled the problem of insufficient semantic alignment in brain-vision decoding methods, which causes reconstruction distortions, by developing a framework that injects fMRI representations into image features. Their results show that incorporating fMRI signals improves accuracy in object detection and segmentation tasks, confirming that fMRI contains rich semantic and spatial cues not fully exploited by current models.
Recent advances in brain-vision decoding have driven significant progress, reconstructing with high fidelity perceived visual stimuli from neural activity, e.g., functional magnetic resonance imaging (fMRI), in the human visual cortex. Most existing methods decode the brain signal using a two-level strategy, i.e., pixel-level and semantic-level. However, these methods rely heavily on low-level pixel alignment yet lack sufficient and fine-grained semantic alignment, resulting in obvious reconstruction distortions of multiple semantic objects. To better understand the brain's visual perception patterns and how current decoding models process semantic objects, we have developed an experimental framework that uses fMRI representations as intervention conditions. By injecting these representations into multi-scale image features via cross-attention, we compare both downstream performance and intermediate feature changes on object detection and instance segmentation tasks with and without fMRI information. Our results demonstrate that incorporating fMRI signals enhances the accuracy of downstream detection and segmentation, confirming that fMRI contains rich multi-object semantic cues and coarse spatial localization information-elements that current models have yet to fully exploit or integrate.