On-Device Qwen2.5: Efficient LLM Inference with Model Compression and Hardware Acceleration
This work addresses the problem of efficient LLM inference for edge computing, offering incremental improvements in compression and speed for a specific hardware setup.
This paper tackles the challenge of deploying large language models on edge devices by presenting an efficient framework for the Qwen2.5-0.5B model on the Xilinx Kria KV260 platform, achieving a 55.08% model compression rate and 5.1 tokens per second output, outperforming a baseline of 2.8 tokens per second.
Transformer-based Large Language Models (LLMs) have significantly advanced AI capabilities but pose considerable challenges for deployment on edge devices due to high computational demands, memory bandwidth constraints, and energy consumption. This paper addresses these challenges by presenting an efficient framework for deploying the Qwen2.5-0.5B model on the Xilinx Kria KV260 edge platform, a heterogeneous system integrating an ARM Cortex-A53 CPU with reconfigurable FPGA logic. Leveraging Activation-aware Weight Quantization (AWQ) with FPGA-accelerated execution pipelines, the proposed approach enhances both model compression rate and system throughput. Additionally, we propose a hybrid execution strategy that intelligently offloads compute-intensive operations to the FPGA while utilizing the CPU for lighter tasks, effectively balancing the computational workload and maximizing overall performance. Our framework achieves a model compression rate of 55.08% compared to the original model and produces output at a rate of 5.1 tokens per second, outperforming the baseline performance of 2.8 tokens per second.