Deep Models, Shallow Alignment: Uncovering the Granularity Mismatch in Neural Decoding
This addresses a fundamental problem in brain-computer interface research for improving neural decoding accuracy, though it is incremental as it builds on existing contrastive learning methods.
The paper tackled the granularity mismatch between deep vision models and neural signals in neural visual decoding by proposing Shallow Alignment, a contrastive learning strategy that aligns neural signals with intermediate representations, resulting in performance gains of 22% to 58% across benchmarks and enabling predictable scaling with model capacity.
Neural visual decoding is a central problem in brain computer interface research, aiming to reconstruct human visual perception and to elucidate the structure of neural representations. However, existing approaches overlook a fundamental granularity mismatch between human and machine vision, where deep vision models emphasize semantic invariance by suppressing local texture information, whereas neural signals preserve an intricate mixture of low-level visual attributes and high-level semantic content. To address this mismatch, we propose Shallow Alignment, a novel contrastive learning strategy that aligns neural signals with intermediate representations of visual encoders rather than their final outputs, thereby striking a better balance between low-level texture details and high-level semantic features. Extensive experiments across multiple benchmarks demonstrate that Shallow Alignment significantly outperforms standard final-layer alignment, with performance gains ranging from 22% to 58% across diverse vision backbones. Notably, our approach effectively unlocks the scaling law in neural visual decoding, enabling decoding performance to scale predictably with the capacity of pre-trained vision backbones. We further conduct systematic empirical analyses to shed light on the mechanisms underlying the observed performance gains.