Exploring The Visual Feature Space for Multimodal Neural Decoding
This work addresses the need for more accurate and detailed neuro-decoding applications, representing an incremental improvement over existing coarse methods.
The paper tackles the problem of imprecise and ambiguous reconstructions in multimodal neural decoding by analyzing vision feature spaces from MLLMs and introducing a zero-shot method, which enhances decoding precision across multiple granularities and is evaluated on a new benchmark with tasks for detailed descriptions and question-answering.
The intrication of brain signals drives research that leverages multimodal AI to align brain modalities with visual and textual data for explainable descriptions. However, most existing studies are limited to coarse interpretations, lacking essential details on object descriptions, locations, attributes, and their relationships. This leads to imprecise and ambiguous reconstructions when using such cues for visual decoding. To address this, we analyze different choices of vision feature spaces from pre-trained visual components within Multimodal Large Language Models (MLLMs) and introduce a zero-shot multimodal brain decoding method that interacts with these models to decode across multiple levels of granularities. % To assess a model's ability to decode fine details from brain signals, we propose the Multi-Granularity Brain Detail Understanding Benchmark (MG-BrainDub). This benchmark includes two key tasks: detailed descriptions and salient question-answering, with metrics highlighting key visual elements like objects, attributes, and relationships. Our approach enhances neural decoding precision and supports more accurate neuro-decoding applications. Code will be available at https://github.com/weihaox/VINDEX.