The Confidence Paradox: Can LLM Know When It's Wrong
This addresses ethical calibration issues in DocVQA for applications requiring reliable and trustworthy AI, though it is incremental as it builds on existing models.
The paper tackles the problem of overconfident and ethically misaligned responses in Document Visual Question Answering (DocVQA) models by proposing HonestVQA, a model-agnostic framework that improves accuracy and F1 by up to 4.3% and reduces overconfidence across multiple datasets.
Document Visual Question Answering (DocVQA) models often produce overconfident or ethically misaligned responses, especially under uncertainty. Existing models like LayoutLMv3, UDOP, and DONUT focus on accuracy but lack ethical calibration. We propose HonestVQA, a model-agnostic, self-supervised framework that aligns model confidence with correctness using weighted loss and contrastive learning. We introduce two new metrics Honesty Score (H-Score) and Ethical Confidence Index (ECI)-to evaluate ethical alignment. HonestVQA improves accuracy and F1 by up to 4.3% across SpDocVQA, InfographicsVQA, and SROIE datasets, while reducing overconfidence. It also generalizes well across domains, achieving 78.9% accuracy and 76.1% F1-score.