BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Models
For practitioners fine-tuning large models, BaLoRA offers a computationally efficient way to obtain both improved accuracy and reliable uncertainty quantification, addressing a key limitation of standard LoRA.
BaLoRA extends LoRA with a Bayesian input-adaptive parameterization, achieving well-calibrated uncertainty estimates and improving prediction accuracy, narrowing the gap with full fine-tuning on NLP and vision tasks. In band gap prediction, it produces zero-shot uncertainty estimates that correlate more strongly with error than a trained LoRA ensemble.
Low-Rank Adaptation (LoRA) has become the standard for fine-tuning large pre-trained models at reduced computational cost. However, its low-rank point-estimate updates limit expressiveness, leave a persistent gap relative to full fine-tuning accuracy, and provide no built-in uncertainty quantification, limiting its applicability in settings where reliability matters as much as accuracy. We introduce BaLoRA, a Bayesian extension of LoRA with a novel input-adaptive Bayesian parameterization of LoRA matrices that adds minimal parameters and compute. Surprisingly, not only does the Bayesian extension yield well-calibrated uncertainty estimates, but the adaptive noise injection underlying our approach also significantly improves prediction accuracy, narrowing the gap with full fine-tuning across both natural language reasoning and vision tasks. When applied to band gap prediction in metal-organic frameworks, BaLoRA produces zero-shot test-time uncertainty estimates that correlate more strongly with model error than a trained ensemble of LoRA models, and improve monotonically with compute without sacrificing accuracy.