Explore Activation Sparsity in Recurrent LLMs for Energy-Efficient Neuromorphic Computing
This work addresses the problem of high energy consumption for deploying LLMs on edge devices, offering an incremental improvement through activation sparsity for neuromorphic computing.
The paper tackles the energy efficiency and latency challenges of deploying Large Language Models (LLMs) on edge devices by proposing a training-free algorithm to sparsify activations in Recurrent LLMs, achieving significant reductions in computational demands and notable energy savings on neuromorphic hardware while maintaining competitive accuracy.
The recent rise of Large Language Models (LLMs) has revolutionized the deep learning field. However, the desire to deploy LLMs on edge devices introduces energy efficiency and latency challenges. Recurrent LLM (R-LLM) architectures have proven effective in mitigating the quadratic complexity of self-attention, making them a potential paradigm for computing on-edge neuromorphic processors. In this work, we propose a low-cost, training-free algorithm to sparsify R-LLMs' activations to enhance energy efficiency on neuromorphic hardware. Our approach capitalizes on the inherent structure of these models, rendering them well-suited for energy-constrained environments. Although primarily designed for R-LLMs, this method can be generalized to other LLM architectures, such as transformers, as demonstrated on the OPT model, achieving comparable sparsity and efficiency improvements. Empirical studies illustrate that our method significantly reduces computational demands while maintaining competitive accuracy across multiple zero-shot learning benchmarks. Additionally, hardware simulations with the SENECA neuromorphic processor underscore notable energy savings and latency improvements. These results pave the way for low-power, real-time neuromorphic deployment of LLMs and demonstrate the feasibility of training-free on-chip adaptation using activation sparsity.