EgoBrain: Synergizing Minds and Eyes For Human Action Understanding
This provides a new paradigm for brain-computer interfaces and human-centered behavior analysis, though it is incremental in applying multimodal AI to EEG-vision fusion.
The authors tackled the problem of human action understanding by creating EgoBrain, the first large-scale dataset synchronizing egocentric vision and EEG, and developed a multimodal learning framework that achieved 66.70% accuracy in action recognition.
The integration of brain-computer interfaces (BCIs), in particular electroencephalography (EEG), with artificial intelligence (AI) has shown tremendous promise in decoding human cognition and behavior from neural signals. In particular, the rise of multimodal AI models have brought new possibilities that have never been imagined before. Here, we present EgoBrain --the world's first large-scale, temporally aligned multimodal dataset that synchronizes egocentric vision and EEG of human brain over extended periods of time, establishing a new paradigm for human-centered behavior analysis. This dataset comprises 61 hours of synchronized 32-channel EEG recordings and first-person video from 40 participants engaged in 29 categories of daily activities. We then developed a muiltimodal learning framework to fuse EEG and vision for action understanding, validated across both cross-subject and cross-environment challenges, achieving an action recognition accuracy of 66.70%. EgoBrain paves the way for a unified framework for brain-computer interface with multiple modalities. All data, tools, and acquisition protocols are openly shared to foster open science in cognitive computing.