LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks
This work addresses the need for efficient uncertainty modeling in machine learning, particularly for large models like transformers, offering a practical solution for real-world decisions requiring calibrated predictions.
The paper tackled the problem of high computational cost and memory footprint in ensemble methods for uncertainty estimation in self-attention networks by introducing LoRA-Ensemble, a parameter-efficient ensembling method based on Low-Rank Adaptation, which outperformed state-of-the-art implicit techniques and matched or exceeded the accuracy of explicit ensembles while achieving superior calibration.
Numerous real-world decisions rely on machine learning algorithms and require calibrated uncertainty estimates. However, modern methods often yield overconfident, uncalibrated predictions. The dominant approach to quantifying the uncertainty inherent in the model is to train an ensemble of separate predictors and measure their empirical variance. In an explicit implementation, the ensemble has high computational cost and memory footprint, especially if the base model itself is already large, like modern transformers. This motivates efforts to develop implicit ensemble methods that emulate the ensemble without explicitly instantiating all its members. We introduce LoRA-Ensemble, a parameter-efficient ensembling method for self-attention networks. It is based on Low-Rank Adaptation (LoRA), originally developed for efficient LLM fine-tuning, and extends it into an implicit ensembling scheme, where all ensemble members share the same, pre-trained self-attention network, but have individual low-rank matrices for the attention projections. The resulting method not only outperforms state-of-the-art implicit techniques like BatchEnsemble, but even matches or exceeds the accuracy of an Explicit Ensemble, while at the same time achieving superior calibration.