UMBRAE: Unified Multimodal Brain Decoding
This addresses challenges in brain-powered research for neuroscience and AI applications, offering a more efficient and generalizable approach, though it appears incremental in building on existing multimodal and alignment techniques.
The authors tackled the problem of accurately decoding spatial information and requiring subject-specific models in brain signal research by proposing UMBRAE, a unified multimodal decoding method that achieves superior results in both new and established tasks, including outperforming subject-specific models with cross-subject training.
We address prevailing challenges of the brain-powered research, departing from the observation that the literature hardly recover accurate spatial information and require subject-specific models. To address these challenges, we propose UMBRAE, a unified multimodal decoding of brain signals. First, to extract instance-level conceptual and spatial details from neural signals, we introduce an efficient universal brain encoder for multimodal-brain alignment and recover object descriptions at multiple levels of granularity from subsequent multimodal large language model (MLLM). Second, we introduce a cross-subject training strategy mapping subject-specific features to a common feature space. This allows a model to be trained on multiple subjects without extra resources, even yielding superior results compared to subject-specific models. Further, we demonstrate this supports weakly-supervised adaptation to new subjects, with only a fraction of the total training data. Experiments demonstrate that UMBRAE not only achieves superior results in the newly introduced tasks but also outperforms methods in well established tasks. To assess our method, we construct and share with the community a comprehensive brain understanding benchmark BrainHub. Our code and benchmark are available at https://weihaox.github.io/UMBRAE.