EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices
This work addresses the problem of efficient on-device inference for sparse LLMs, enabling better privacy and availability on edge devices, though it is incremental as it builds on existing MoE architectures with optimizations.
The paper tackles the challenge of deploying large sparse mixture-of-expert (MoE) language models on mobile devices by proposing EdgeMoE, an on-device inference engine that partitions model weights across storage hierarchies and uses techniques like expert-wise bitwidth adaptation and preloading, achieving significant memory savings and speedup over baselines.
Large language models (LLMs) such as GPTs and Mixtral-8x7B have revolutionized machine intelligence due to their exceptional abilities in generic ML tasks. Transiting LLMs from datacenters to edge devices brings benefits like better privacy and availability, but is challenged by their massive parameter size and thus unbearable runtime costs. To this end, we present EdgeMoE, an on-device inference engine for mixture-of-expert (MoE) LLMs -- a popular form of sparse LLM that scales its parameter size with almost constant computing complexity. EdgeMoE achieves both memory- and compute-efficiency by partitioning the model into the storage hierarchy: non-expert weights are held in device memory; while expert weights are held on external storage and fetched to memory only when activated. This design is motivated by a key observation that expert weights are bulky but infrequently used due to sparse activation. To further reduce the expert I/O swapping overhead, EdgeMoE incorporates two novel techniques: (1) expert-wise bitwidth adaptation that reduces the expert sizes with tolerable accuracy loss; (2) expert preloading that predicts the activated experts ahead of time and preloads it with the compute-I/O pipeline. On popular MoE LLMs and edge devices, EdgeMoE showcase significant memory savings and speedup over competitive baselines. The code is available at https://github.com/UbiquitousLearning/mllm.