Latent Confidence Alignment for LLM Self-Assessment
For practitioners needing reliable LLM confidence estimates, this work offers a new metric that disentangles model ability from item difficulty, though it is domain-specific and incremental.
The paper proposes Latent Confidence Alignment Error (LCAE) to measure consistency between LLM self-assessment and latent error probability, improving self-assessment quality without affecting model ability on a medical-domain dataset with 20 models.
Confidence calibration in large language models (LLMs) is commonly evaluated by comparing predicted confidence with observed accuracy. However, such approaches do not model item difficulty, making it difficult to interpret discrepancies and to determine whether model confidence reflects genuine self-assessment or is merely a byproduct of the response generation process. To address this, we adopt a Rasch model-based latent ability framework and a metacognitive perspective, and propose Latent Confidence Alignment Error (LCAE) to measure the consistency between model self-assessment and the latent error probability implied by model ability and item difficulty. We further incorporate item difficulty as an external signal with a reasoning mechanism. Experiments on a medical-domain dataset with 20 models show that the proposed approach improves self-assessment quality without affecting model ability, and reveals an association between reliability and inference cost.