Brainformer: Mimic Human Visual Brain Functions to Machine Vision Models via fMRI
This work addresses the challenge of enhancing neural networks by transferring knowledge from human perception, potentially benefiting AI and neuroscience, though it appears incremental as it builds on existing Transformer architectures.
The authors tackled the problem of improving machine vision models by mimicking human brain functions using fMRI data, introducing Brainformer, a Transformer-based framework with novel components like 3D Voxels Embedding and Brain fMRI Guidance Loss, and achieved results comparable to State-of-the-Art methods in image recognition tasks.
Human perception plays a vital role in forming beliefs and understanding reality. A deeper understanding of brain functionality will lead to the development of novel deep neural networks. In this work, we introduce a novel framework named Brainformer, a straightforward yet effective Transformer-based framework, to analyze Functional Magnetic Resonance Imaging (fMRI) patterns in the human perception system from a machine-learning perspective. Specifically, we present the Multi-scale fMRI Transformer to explore brain activity patterns through fMRI signals. This architecture includes a simple yet efficient module for high-dimensional fMRI signal encoding and incorporates a novel embedding technique called 3D Voxels Embedding. Secondly, drawing inspiration from the functionality of the brain's Region of Interest, we introduce a novel loss function called Brain fMRI Guidance Loss. This loss function mimics brain activity patterns from these regions in the deep neural network using fMRI data. This work introduces a prospective approach to transferring knowledge from human perception to neural networks. Our experiments demonstrate that leveraging fMRI information allows the machine vision model to achieve results comparable to State-of-the-Art methods in various image recognition tasks.