CM1 -- A Dataset for Evaluating Few-Shot Information Extraction with Large Vision Language Models
This work addresses the challenge of digitizing archives with limited annotated data, though it is incremental as it focuses on dataset creation and benchmarking.
The authors tackled the problem of evaluating few-shot information extraction from handwritten documents by introducing the CM1 dataset, a historic collection of forms, and found that large vision language models outperform traditional full-page extraction models when only a few training samples are available.
The automatic extraction of key-value information from handwritten documents is a key challenge in document analysis. A reliable extraction is a prerequisite for the mass digitization efforts of many archives. Large Vision Language Models (LVLM) are a promising technology to tackle this problem especially in scenarios where little annotated training data is available. In this work, we present a novel dataset specifically designed to evaluate the few-shot capabilities of LVLMs. The CM1 documents are a historic collection of forms with handwritten entries created in Europe to administer the Care and Maintenance program after World War Two. The dataset establishes three benchmarks on extracting name and birthdate information and, furthermore, considers different training set sizes. We provide baseline results for two different LVLMs and compare performances to an established full-page extraction model. While the traditional full-page model achieves highly competitive performances, our experiments show that when only a few training samples are available the considered LVLMs benefit from their size and heavy pretraining and outperform the classical approach.