Information Redundancy and Biases in Public Document Information Extraction Benchmarks
This work addresses biases in benchmarks for document information extraction, which is crucial for researchers and practitioners to assess model generalization accurately, though it is incremental as it focuses on improving existing evaluation methods.
The study identified significant information redundancy in public benchmarks for Key-Information Extraction, with 75% template replication in SROIE and 16% in FUNSD, and proposed resampling strategies to better evaluate model generalization, showing performance drops of up to 10.5% F1 for less-suited models.
Advances in the Visually-rich Document Understanding (VrDU) field and particularly the Key-Information Extraction (KIE) task are marked with the emergence of efficient Transformer-based approaches such as the LayoutLM models. Despite the good performance of KIE models when fine-tuned on public benchmarks, they still struggle to generalize on complex real-life use-cases lacking sufficient document annotations. Our research highlighted that KIE standard benchmarks such as SROIE and FUNSD contain significant similarity between training and testing documents and can be adjusted to better evaluate the generalization of models. In this work, we designed experiments to quantify the information redundancy in public benchmarks, revealing a 75% template replication in SROIE official test set and 16% in FUNSD. We also proposed resampling strategies to provide benchmarks more representative of the generalization ability of models. We showed that models not suited for document analysis struggle on the adjusted splits dropping on average 10,5% F1 score on SROIE and 3.5% on FUNSD compared to multi-modal models dropping only 7,5% F1 on SROIE and 0.5% F1 on FUNSD.