Reservoir Computing inspired Matrix Multiplication-free Language Model
This work addresses computational efficiency for natural language processing applications, but it is incremental as it builds on existing matrix multiplication-free models with minor architectural tweaks.
The study tackled the high computational cost of large language models by proposing a matrix multiplication-free language model with reservoir computing-inspired architecture, resulting in up to 19% fewer parameters, 9.9% faster training, and 8.0% faster inference while maintaining comparable performance.
Large language models (LLMs) have achieved state-of-the-art performance in natural language processing; however, their high computational cost remains a major bottleneck. In this study, we target computational efficiency by focusing on a matrix multiplication free language model (MatMul-free LM) and further reducing the training cost through an architecture inspired by reservoir computing. Specifically, we partially fix and share the weights of selected layers in the MatMul-free LM and insert reservoir layers to obtain rich dynamic representations without additional training overhead. Additionally, several operations are combined to reduce memory accesses. Experimental results show that the proposed architecture reduces the number of parameters by up to 19%, training time by 9.9%, and inference time by 8.0%, while maintaining comparable performance to the baseline model.