RWKV-edge: Deeply Compressed RWKV for Resource-Constrained Devices
This work addresses deployment challenges for LLMs on mobile robots and smartphones, representing an incremental improvement through compression tailored to the RWKV architecture.
The paper tackles the problem of deploying large language models (RWKV) on resource-constrained devices by proposing a suite of compression techniques, reducing memory footprint by 3.4x to 5x with negligible accuracy degradation and requiring 4x less memory than transformer LLMs with similar accuracy.
To deploy LLMs on resource-contained platforms such as mobile robots and smartphones, non-transformers LLMs have achieved major breakthroughs. Recently, a novel RNN-based LLM family, Repentance Weighted Key Value (RWKV) has shown strong computational efficiency; nevertheless, RWKV models still have high parameter counts which limited their deployment. In this paper, we propose a suite of compression techniques, ranging from model architecture optimizations to post-training compression, tailored to the RWKV architecture. Combined, our techniques reduce the memory footprint of RWKV models by 3.4x -- 5x with only negligible degradation in accuracy; compared to transformer LLMs with similar accuracy, our models require 4x less memory footprint.