A Systematic Evaluation of Sample-Level Tokenization Strategies for MEG Foundation Models
This work addresses the problem of tokenization for neural foundation models in neuroimaging, providing practical guidance for researchers, but it is incremental as it compares existing strategies rather than introducing a breakthrough.
The study systematically evaluated sample-level tokenization strategies for transformer-based large neuroimaging models on MEG data, finding that both learnable and non-learnable methods achieved high reconstruction accuracy and comparable performance across most criteria, indicating simple fixed tokenization can be effective.
Recent success in natural language processing has motivated growing interest in large-scale foundation models for neuroimaging data. Such models often require discretization of continuous neural time series data, a process referred to as 'tokenization'. However, the impact of different tokenization strategies for neural data is currently poorly understood. In this work, we present a systematic evaluation of sample-level tokenization strategies for transformer-based large neuroimaging models (LNMs) applied to magnetoencephalography (MEG) data. We compare learnable and non-learnable tokenizers by examining their signal reconstruction fidelity and their impact on subsequent foundation modeling performance (token prediction, biological plausibility of generated data, preservation of subject-specific information, and performance on downstream tasks). For the learnable tokenizer, we introduce a novel approach based on an autoencoder. Experiments were conducted on three publicly available MEG datasets spanning different acquisition sites, scanners, and experimental paradigms. Our results show that both learnable and non-learnable discretization schemes achieve high reconstruction accuracy and broadly comparable performance across most evaluation criteria, suggesting that simple fixed sample-level tokenization strategies can be used in the development of neural foundation models. The code is available at https://github.com/OHBA-analysis/Cho2026_Tokenizer.