MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices
This addresses the challenge of efficient LLM deployment on edge devices for mobile users, representing an incremental improvement in optimization techniques.
The paper tackles the problem of high computational resource consumption for large language model (LLM) inference on mobile devices by introducing MNN-LLM, a framework that uses model quantization, DRAM-Flash hybrid storage, and optimization strategies, achieving up to an 8.6x speed increase compared to existing frameworks.
Large language models (LLMs) have demonstrated exceptional performance across a variety of tasks. However, their substantial scale leads to significant computational resource consumption during inference, resulting in high costs. Consequently, edge device inference presents a promising solution. The primary challenges of edge inference include memory usage and inference speed. This paper introduces MNN-LLM, a framework specifically designed to accelerate the deployment of large language models on mobile devices. MNN-LLM addresses the runtime characteristics of LLMs through model quantization and DRAM-Flash hybrid storage, effectively reducing memory usage. It rearranges weights and inputs based on mobile CPU instruction sets and GPU characteristics while employing strategies such as multicore load balancing, mixed-precision floating-point operations, and geometric computations to enhance performance. Notably, MNN-LLM achieves up to a 8.6x speed increase compared to current mainstream LLM-specific frameworks.