HYCEDIS: HYbrid Confidence Engine for Deep Document Intelligence System
This addresses the need for reliable confidence scores to safely deploy AI in industrial document systems, offering a significant improvement over existing methods.
The paper tackles the problem of measuring confidence for deep-learning-based information extraction from scanned documents, proposing a novel architecture that outperforms competing estimators by a large margin and generalizes to out-of-distribution data.
Measuring the confidence of AI models is critical for safely deploying AI in real-world industrial systems. One important application of confidence measurement is information extraction from scanned documents. However, there exists no solution to provide reliable confidence score for current state-of-the-art deep-learning-based information extractors. In this paper, we propose a complete and novel architecture to measure confidence of current deep learning models in document information extraction task. Our architecture consists of a Multi-modal Conformal Predictor and a Variational Cluster-oriented Anomaly Detector, trained to faithfully estimate its confidence on its outputs without the need of host models modification. We evaluate our architecture on real-wold datasets, not only outperforming competing confidence estimators by a huge margin but also demonstrating generalization ability to out-of-distribution data.