Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices
For practitioners needing to fine-tune LLMs on consumer hardware, this work provides practical memory reduction techniques without sacrificing model quality.
This paper introduces a suite of techniques (quantization, checkpointing, softmax approximation, logits masking) to reduce peak memory during LoRA fine-tuning of LLMs, achieving up to 26x and 28x memory reduction on Llama-3.2 3B and Qwen-2.5 3B respectively, enabling fine-tuning on edge devices.
Fine-tuning of Large Language Models (LLMs) using Low-Rank Adaptation (LoRA) on an end-user's data offers personalized experiences while keeping data private, but faces severe memory constraints on consumer hardware. Peak memory during fine-tuning often exceeds device limits, especially for models with billions of parameters and long-context training data. This paper introduces a suite of complementary techniques to reduce memory footprint without sacrificing model quality: (1) base model quantization with on-the-fly dequantization, (2) memory-efficient checkpointing combining selective activation caching and disk offloading, (3) softmax approximation using semantically relevant token subsets, and (4) logits masking. Experiments on Llama-3.2 3B and Qwen-2.5 3B demonstrate up to $26\times$ and $28\times$ reduction in peak memory, enabling fine-tuning on resource-constrained devices.