SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking
This work addresses energy efficiency for AI practitioners by introducing a novel spiking approach to LLMs, though it appears incremental as it builds on existing quantization methods.
The authors tackled the high energy and computational demands of large language models (LLMs) by redesigning 7-70 billion parameter LLMs using bio-plausible spiking mechanisms, resulting in SpikeLLM, which reduced WikiText2 perplexity by 11.01% and improved reasoning accuracy by 2.55% on a LLAMA-7B model.
Recent advancements in large language models (LLMs) with billions of parameters have improved performance in various applications, but their inference processes demand significant energy and computational resources. In contrast, the human brain, with approximately 86 billion neurons, is much more energy-efficient than LLMs with similar parameters. Inspired by this, we redesign 7$\sim$70 billion parameter LLMs using bio-plausible spiking mechanisms, emulating the efficient behavior of the human brain. We propose the first spiking large language model, SpikeLLM. Coupled with the proposed model, two essential approaches are proposed to improve spike training efficiency: Generalized Integrate-and-Fire (GIF) neurons to compress spike length from $T$ to $\frac{T}{L} \log_2 L$ bits, and an Optimal Brain Spiking framework to divide outlier channels and allocate different $T$ for GIF neurons, which further compresses spike length to approximate $log_2T$ bits. The necessity of spike-driven LLM is proved by comparison with quantized LLMs with similar operations. In the OmniQuant pipeline, SpikeLLM reduces 11.01% WikiText2 perplexity and improves 2.55% accuracy of common scene reasoning on a LLAMA-7B W4A4 model. In the GPTQ pipeline, SpikeLLM achieves direct additive in linear layers, significantly exceeding PB-LLMs.