Overcoming the Domain Gap in Neural Action Representations
This addresses the challenge of building robust brain-machine interfaces by enabling models to work on unlabeled subjects, though it appears incremental as it builds on existing methods for domain adaptation.
The paper tackled the domain gap problem in neural action representations across individuals by using 3D pose data and cross-animal data swapping during training, achieving improved generalization on three multimodal datasets including flies, human ECoG, and RGB video.
Relating animal behaviors to brain activity is a fundamental goal in neuroscience, with practical applications in building robust brain-machine interfaces. However, the domain gap between individuals is a major issue that prevents the training of general models that work on unlabeled subjects. Since 3D pose data can now be reliably extracted from multi-view video sequences without manual intervention, we propose to use it to guide the encoding of neural action representations together with a set of neural and behavioral augmentations exploiting the properties of microscopy imaging. To reduce the domain gap, during training, we swap neural and behavioral data across animals that seem to be performing similar actions. To demonstrate this, we test our methods on three very different multimodal datasets; one that features flies and their neural activity, one that contains human neural Electrocorticography (ECoG) data, and lastly the RGB video data of human activities from different viewpoints.