On the Robustness of Verbal Confidence of LLMs in Adversarial Attacks
This addresses a critical safety issue for deploying LLMs in high-stakes applications, but it is incremental as it builds on existing adversarial attack research.
The paper tackled the problem of ensuring robust verbal confidence in large language models (LLMs) under adversarial attacks, showing that attacks can significantly jeopardize confidence estimates and lead to frequent answer changes, with current methods being vulnerable and defenses ineffective.
Robust verbal confidence generated by large language models (LLMs) is crucial for the deployment of LLMs to ensure transparency, trust, and safety in human-AI interactions across many high-stakes applications. In this paper, we present the first comprehensive study on the robustness of verbal confidence under adversarial attacks. We introduce a novel framework for attacking verbal confidence scores through both perturbation and jailbreak-based methods, and show that these attacks can significantly jeopardize verbal confidence estimates and lead to frequent answer changes. We examine a variety of prompting strategies, model sizes, and application domains, revealing that current confidence elicitation methods are vulnerable and that commonly used defence techniques are largely ineffective or counterproductive. Our findings underscore the urgent need to design more robust mechanisms for confidence expression in LLMs, as even subtle semantic-preserving modifications can lead to misleading confidence in responses.