LLMPi: Optimizing LLMs for High-Throughput on Raspberry Pi
This work addresses the problem of efficient AI deployment on low-power embedded systems for mobile and embedded applications, but it is incremental as it builds on existing quantization methods.
The paper tackles deploying Large Language Models on resource-constrained edge devices like Raspberry Pi by using quantization techniques, resulting in significant reductions in energy consumption while maintaining inference quality for real-time conversational AI.
Deploying Large Language Models (LLMs) on resource-constrained edge devices like the Raspberry Pi presents challenges in computational efficiency, power consumption, and response latency. This paper explores quantization-based optimization techniques to enable high-throughput, energy-efficient execution of LLMs on low-power embedded systems. Our approach leverages k-quantization, a Post-Training Quantization (PTQ) method designed for different bit-widths, enabling efficient 2-bit, 4-bit, 6-bit, and 8-bit weight quantization. Additionally, we employ ternary quantization using Quantization-Aware Training (QAT) for BitNet models, allowing for more effective adaptation to lower-bit representations while preserving accuracy. Our findings highlight the potential of quantized LLMs for real-time conversational AI on edge devices, paving the way for low-power, high-efficiency AI deployment in mobile and embedded applications. This study demonstrates that aggressive quantization strategies can significantly reduce energy consumption while maintaining inference quality, making LLMs practical for resource-limited environments.