Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model
This work addresses the problem of energy-efficient language model deployment at the edge for applications requiring privacy and resource constraints, representing an incremental improvement by optimizing existing methods for hardware compatibility.
The authors tackled the challenge of implementing transformer-based language models on neuromorphic hardware by introducing Sorbet, a spiking language model that replaces energy-intensive operations like softmax and layer normalization with novel shifting-based alternatives, achieving 27.16× energy savings compared to BERT while maintaining competitive performance on the GLUE benchmark.
For reasons such as privacy, there are use cases for language models at the edge. This has given rise to small language models targeted for deployment in resource-constrained devices where energy efficiency is critical. Spiking neural networks (SNNs) offer a promising solution due to their energy efficiency, and there are already works on realizing transformer-based models on SNNs. However, key operations like softmax and layer normalization (LN) are difficult to implement on neuromorphic hardware, and many of these early works sidestepped them. To address these challenges, we introduce Sorbet, a transformer-based spiking language model that is more neuromorphic hardware-compatible. Sorbet incorporates a novel shifting-based softmax called PTsoftmax and a Bit Shifting PowerNorm (BSPN), both designed to replace the respective energy-intensive operations. By leveraging knowledge distillation and model quantization, Sorbet achieved a highly compressed binary weight model that maintains competitive performance while achieving $27.16\times$ energy savings compared to BERT. We validate Sorbet through extensive testing on the GLUE benchmark and a series of ablation studies, demonstrating its potential as an energy-efficient solution for language model inference. Our code is publicly available at \href{https://github.com/Kaiwen-Tang/Sorbet}{https://github.com/Kaiwen-Tang/Sorbet}