Crayon: Customized On-Device LLM via Instant Adapter Blending and Edge-Server Hybrid Inference
This addresses memory, computational, and privacy issues for users needing personalized LLMs, but it is incremental as it builds on existing adapter and hybrid inference methods.
The paper tackles the problem of customizing large language models (LLMs) for on-device use to reduce cloud overhead and privacy risks, proposing Crayon, which blends adapters without extra training and uses hybrid inference, showing efficacy on a novel benchmark.
The customization of large language models (LLMs) for user-specified tasks gets important. However, maintaining all the customized LLMs on cloud servers incurs substantial memory and computational overheads, and uploading user data can also lead to privacy concerns. On-device LLMs can offer a promising solution by mitigating these issues. Yet, the performance of on-device LLMs is inherently constrained by the limitations of small-scaled models. To overcome these restrictions, we first propose Crayon, a novel approach for on-device LLM customization. Crayon begins by constructing a pool of diverse base adapters, and then we instantly blend them into a customized adapter without extra training. In addition, we develop a device-server hybrid inference strategy, which deftly allocates more demanding queries or non-customized tasks to a larger, more capable LLM on a server. This ensures optimal performance without sacrificing the benefits of on-device customization. We carefully craft a novel benchmark from multiple question-answer datasets, and show the efficacy of our method in the LLM customization.