Justified or Just Convincing? Error Verifiability as a Dimension of LLM Quality
This addresses the need for reliable error detection in high-stakes LLM deployments, establishing error verifiability as a distinct quality dimension that requires dedicated methods, though it is incremental in proposing specific metrics and interventions.
The paper tackles the problem of evaluating whether LLM-generated justifications help users distinguish correct from incorrect answers, formalizing this as error verifiability and proposing a balanced metric validated with high human agreement. It finds that common approaches like post-training and scaling do not improve verifiability, but introduces two domain-specific methods that succeed, such as reflect-and-rephrase for mathematical reasoning, which improve verifiability by incorporating external information.
As LLMs are deployed in high-stakes settings, users must judge the correctness of individual responses, often relying on model-generated justifications such as reasoning chains or explanations. Yet, no standard measure exists for whether these justifications help users distinguish correct answers from incorrect ones. We formalize this idea as error verifiability and propose $v_{\text{bal}}$, a balanced metric that measures whether justifications enable raters to accurately assess answer correctness, validated against human raters who show high agreement. We find that neither common approaches, such as post-training and model scaling, nor more targeted interventions recommended improve verifiability. We introduce two methods that succeed at improving verifiability: reflect-and-rephrase (RR) for mathematical reasoning and oracle-rephrase (OR) for factual QA, both of which improve verifiability by incorporating domain-appropriate external information. Together, our results establish error verifiability as a distinct dimension of response quality that does not emerge from accuracy improvements alone and requires dedicated, domain-aware methods to address.