Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation
For practitioners deploying LLMs, this provides a lightweight method to mitigate overconfidence during fine-tuning, addressing a key trustworthiness bottleneck.
DALorRA introduces a variational Bayesian sparse framework for LoRA that calibrates LLM uncertainty by stochastically masking rank dimensions, achieving excellent calibration without sacrificing reasoning accuracy.
Large language models (LLMs) exhibit remarkable reasoning capabilities, but their task-specific fine-tuning is notoriously plagued by overconfidence, severely hindering trustworthy deployment. We propose Data-Adaptive Lower-Rank Adaptation (DALorRA), a simple and effective variational Bayesian sparse framework that shifts the paradigm of uncertainty quantification from the dense parameter space to the lightweight rank level of low-rank adaptation (LoRA). With the insight that LoRA essentially aggregates multiple rank-one components that may provide superfluous model capacity, DALorRA imposes stochastic masking on rank dimensions, enabling Bayesian regularization of model capacity during training and ensemble-like calibration during inference. Extensive experiments demonstrate DALorRA's excellent calibration of LLMs without compromising reasoning accuracy.