Leveraging Semantic Segmentation Masks with Embeddings for Fine-Grained Form Classification
This work addresses the labor-intensive and error-prone manual classification of historical documents for fields like genealogy and legal research, though it is incremental in combining existing methods.
The paper tackles fine-grained form classification of historical documents by integrating semantic segmentation with deep learning embeddings, achieving improved clustering and classification results on novel French 19th-century and U.S. 1950 Census datasets.
Efficient categorization of historical documents is crucial for fields such as genealogy, legal research, and historical scholarship, where manual classification is impractical for large collections due to its labor-intensive and error-prone nature. To address this, we propose a representational learning strategy that integrates semantic segmentation and deep learning models such as ResNet, CLIP, Document Image Transformer (DiT), and masked auto-encoders (MAE), to generate embeddings that capture document features without predefined labels. To the best of our knowledge, we are the first to evaluate embeddings on fine-grained, unsupervised form classification. To improve these embeddings, we propose to first employ semantic segmentation as a preprocessing step. We contribute two novel datasets$\unicode{x2014}$the French 19th-century and U.S. 1950 Census records$\unicode{x2014}$to demonstrate our approach. Our results show the effectiveness of these various embedding techniques in distinguishing similar document types and indicate that applying semantic segmentation can greatly improve clustering and classification results. The census datasets are available at https://github.com/tahlor/census_forms