Scalable Machines with Intrinsic Higher Mental-State Dynamics
This work addresses scalability issues in large-scale AI models for researchers and practitioners, though it appears incremental as it builds on existing Transformer architectures with biological insights.
The paper tackles the problem of inefficient attention mechanisms in Transformers by introducing a biologically-inspired method that pre-selects relevant information before attention, resulting in faster learning and reduced computational demand on ImageNet-1K compared to a standard Vision Transformer.
Drawing on recent breakthroughs in cellular neurobiology and detailed biophysical modeling linking neocortical pyramidal neurons to distinct mental-state regimes, this work introduces a mathematically grounded formulation showing how models (e.g., Transformers) can implement computational principles underlying awake imaginative thought to pre-select relevant information before attention is applied via triadic modulation loops among queries ($Q$), keys ($K$), and values ($V$).~Scalability experiments on ImageNet-1K, benchmarked against a standard Vision Transformer (ViT), demonstrate significantly faster learning with reduced computational demand (fewer heads, layers, and tokens), consistent with our prior findings in reinforcement learning and language modeling. The approach operates at approximately $\mathcal{O}(N)$ complexity with respect to the number of input tokens $N$.