PIPO: Pipelined Offloading for Efficient Inference on Consumer Devices
This addresses the challenge of deploying LLMs on resource-constrained consumer devices, offering a significant efficiency improvement over existing methods.
The paper tackled the problem of low GPU utilization in offloading large language models to consumer devices by proposing PIPO, a pipelined offloading framework, which increased GPU utilization from below 40% to over 90% and achieved up to 3.1× higher throughput on a laptop with a 6GB GPU.
The high memory and computation demand of large language models (LLMs) makes them challenging to be deployed on consumer devices due to limited GPU memory. Offloading can mitigate the memory constraint but often suffers from low GPU utilization, leading to low inference efficiency. In this work, we propose a novel framework, called pipelined offloading (PIPO), for efficient inference on consumer devices. PIPO designs a fine-grained offloading pipeline, complemented with optimized data transfer and computation, to achieve high concurrency and efficient scheduling for inference. Experimental results show that compared with state-of-the-art baseline, PIPO increases GPU utilization from below 40% to over 90% and achieves up to 3.1$\times$ higher throughput, running on a laptop equipped with a RTX3060 GPU of 6GB memory.