Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models
This work addresses calibration and generalization issues in LLMs for NLP applications, presenting an incremental improvement through a simple combination of existing techniques.
The paper tackles the problem of overconfidence and poor calibration in fine-tuned Large Language Models (LLMs) on small datasets by combining Low-Rank Adaptation (LoRA) with Gaussian Stochastic Weight Averaging (SWAG) for approximate Bayesian inference, resulting in improved generalization, calibration, and robustness against distribution shift across NLP benchmarks.
Fine-tuned Large Language Models (LLMs) often suffer from overconfidence and poor calibration, particularly when fine-tuned on small datasets. To address these challenges, we propose a simple combination of Low-Rank Adaptation (LoRA) with Gaussian Stochastic Weight Averaging (SWAG), facilitating approximate Bayesian inference in LLMs. Through extensive testing across several Natural Language Processing (NLP) benchmarks, we demonstrate that our straightforward and computationally efficient approach improves model generalization and calibration competitively with comparable, more sophisticated methods for Bayesian inference in LLMs. We further show that our method exhibits greater robustness against distribution shift, as reflected in its improved performance on out-of-distribution tasks.