The Effects of Data Augmentation on Confidence Estimation for LLMs
This addresses the problem of unreliable confidence estimates in closed-source LLMs for users needing trustworthy model outputs, though it appears incremental as it builds on existing augmentation techniques.
The paper studied how different data augmentation methods affect confidence estimation in large language models, finding that greater data diversity improves performance and reduces overconfidence, with random combination of augmentations being a promising approach.
Confidence estimation is crucial for reflecting the reliability of large language models (LLMs), particularly in the widely used closed-source models. Utilizing data augmentation for confidence estimation is viable, but discussions focus on specific augmentation techniques, limiting its potential. We study the impact of different data augmentation methods on confidence estimation. Our findings indicate that data augmentation strategies can achieve better performance and mitigate the impact of overconfidence. We investigate the influential factors related to this and discover that, while preserving semantic information, greater data diversity enhances the effectiveness of augmentation. Furthermore, the impact of different augmentation strategies varies across different range of application. Considering parameter transferability and usability, the random combination of augmentations is a promising choice.