Towards 3D Acceleration for low-power Mixture-of-Experts and Multi-Head Attention Spiking Transformers
This addresses the problem of energy-efficient hardware for brain-inspired AI accelerators, representing an incremental advance in specialized hardware design.
The paper tackles the lack of hardware support for spiking transformers by introducing the first 3D hardware architecture for Mixture-of-Experts and Multi-Head Attention spiking transformers, demonstrating significant optimization of energy efficiency and latency compared to conventional 2D CMOS integration.
Spiking Neural Networks(SNNs) provide a brain-inspired and event-driven mechanism that is believed to be critical to unlock energy-efficient deep learning. The mixture-of-experts approach mirrors the parallel distributed processing of nervous systems, introducing conditional computation policies and expanding model capacity without scaling up the number of computational operations. Additionally, spiking mixture-of-experts self-attention mechanisms enhance representation capacity, effectively capturing diverse patterns of entities and dependencies between visual or linguistic tokens. However, there is currently a lack of hardware support for highly parallel distributed processing needed by spiking transformers, which embody a brain-inspired computation. This paper introduces the first 3D hardware architecture and design methodology for Mixture-of-Experts and Multi-Head Attention spiking transformers. By leveraging 3D integration with memory-on-logic and logic-on-logic stacking, we explore such brain-inspired accelerators with spatially stackable circuitry, demonstrating significant optimization of energy efficiency and latency compared to conventional 2D CMOS integration.