Evaluating Uncertainty Quantification Methods in Argumentative Large Language Models
This work addresses reliability in explainable AI for decision-making, but it is incremental as it applies existing UQ methods to a specific domain.
The study tackled the problem of evaluating uncertainty quantification methods in argumentative large language models for claim verification tasks, finding that direct prompting outperformed more complex approaches.
Research in uncertainty quantification (UQ) for large language models (LLMs) is increasingly important towards guaranteeing the reliability of this groundbreaking technology. We explore the integration of LLM UQ methods in argumentative LLMs (ArgLLMs), an explainable LLM framework for decision-making based on computational argumentation in which UQ plays a critical role. We conduct experiments to evaluate ArgLLMs' performance on claim verification tasks when using different LLM UQ methods, inherently performing an assessment of the UQ methods' effectiveness. Moreover, the experimental procedure itself is a novel way of evaluating the effectiveness of UQ methods, especially when intricate and potentially contentious statements are present. Our results demonstrate that, despite its simplicity, direct prompting is an effective UQ strategy in ArgLLMs, outperforming considerably more complex approaches.