Variational Low-Rank Adaptation Using IVON
This work addresses the challenge of efficient and accurate fine-tuning of large language models for practitioners, though it is incremental as it builds on existing LoRA and IVON methods.
The paper tackles the problem of improving accuracy and calibration in Low-Rank Adaptation (LoRA) for large language models by replacing AdamW with the Improved Variational Online Newton (IVON) algorithm, resulting in a 2.8% accuracy improvement and 4.6% reduction in expected calibration error for Llama-2 with 7 billion parameters.
We show that variational learning can significantly improve the accuracy and calibration of Low-Rank Adaptation (LoRA) without a substantial increase in the cost. We replace AdamW by the Improved Variational Online Newton (IVON) algorithm to finetune large language models. For Llama-2 with 7 billion parameters, IVON improves the accuracy over AdamW by 2.8% and expected calibration error by 4.6%. The accuracy is also better than the other Bayesian alternatives, yet the cost is lower and the implementation is easier. Our work provides additional evidence for the effectiveness of IVON for large language models. The code is available at https://github.com/team-approx-bayes/ivon-lora.