BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models
This work addresses the memory efficiency challenge for researchers and practitioners finetuning large models on limited hardware, representing an incremental improvement over existing optimization methods.
The paper tackles the problem of memory-intensive full parameter finetuning for large language models by proposing BAdam, a method that reduces memory usage while maintaining or improving performance, achieving results comparable to Adam and outperforming baselines like LoRA on benchmarks such as MT-bench and math tasks.
This work presents BAdam, an optimization method that leverages the block coordinate descent (BCD) framework with Adam's update rule. BAdam offers a memory efficient approach to the full parameter finetuning of large language models. We conduct a theoretical convergence analysis for BAdam in the deterministic case. Experimentally, we apply BAdam to finetune the Llama 3-8B and Llama 3-70B models using a single RTX3090-24GB GPU and 4 A100-80GB GPUs, respectively. The results confirm BAdam's efficiency in terms of memory usage, running time, and optimization capability. Furthermore, the downstream performance evaluation based on MT-bench and math benchmarks shows that BAdam outperforms existing memory efficient baselines such as LoRA. It also demonstrates that BAdam can achieve comparable or even superior performance compared to Adam. Finally, the ablation study using SGD's update rule illustrates the suitability of BCD for finetuning LLMs. Our code can be easily integrated into any PyTorch-based codebase and is available at https://github.com/Ledzy/BAdam.