Visual FUDGE: Form Understanding via Dynamic Graph Editing
This addresses form understanding for degraded, historical, or resource-poor language forms where text recognition is difficult, offering a practical alternative to language model-based methods.
The paper tackles form understanding by predicting text entities and their relationships in form images using a graph-based model that iteratively edits graph structure, achieving nearly the same entity linking performance as large-scale language models on the FUNSD dataset while relying only on visual features from a small training set, and it is state-of-the-art on the historical NAF dataset.
We address the problem of form understanding: finding text entities and the relationships/links between them in form images. The proposed FUDGE model formulates this problem on a graph of text elements (the vertices) and uses a Graph Convolutional Network to predict changes to the graph. The initial vertices are detected text lines and do not necessarily correspond to the final text entities, which can span multiple lines. Also, initial edges contain many false-positive relationships. FUDGE edits the graph structure by combining text segments (graph vertices) and pruning edges in an iterative fashion to obtain the final text entities and relationships. While recent work in this area has focused on leveraging large-scale pre-trained Language Models (LM), FUDGE achieves almost the same level of entity linking performance on the FUNSD dataset by learning only visual features from the (small) provided training set. FUDGE can be applied on forms where text recognition is difficult (e.g. degraded or historical forms) and on forms in resource-poor languages where pre-training such LMs is challenging. FUDGE is state-of-the-art on the historical NAF dataset.