Bootstrapping Weakly Supervised Segmentation-free Word Spotting through HMM-based Alignment
This reduces annotation effort for word spotting in handwritten documents, enabling broader application to collections with existing transcripts, though it is incremental as it builds on existing segmentation-free methods.
The paper tackles the problem of training word spotting models without bounding box annotations by using transcripts and a hidden Markov model-based alignment to automatically generate weakly annotated data, achieving performance within a few mAP% of fully supervised models using only 1-7% of annotated data.
Recent work in word spotting in handwritten documents has yielded impressive results. This progress has largely been made by supervised learning systems, which are dependent on manually annotated data, making deployment to new collections a significant effort. In this paper, we propose an approach that utilises transcripts without bounding box annotations to train segmentation-free query-by-string word spotting models, given a partially trained model. This is done through a training-free alignment procedure based on hidden Markov models. This procedure creates a tentative mapping between word region proposals and the transcriptions to automatically create additional weakly annotated training data, without choosing any single alignment possibility as the correct one. When only using between 1% and 7% of the fully annotated training sets for partial convergence, we automatically annotate the remaining training data and successfully train using it. On all our datasets, our final trained model then comes within a few mAP% of the performance from a model trained with the full training set used as ground truth. We believe that this will be a significant advance towards a more general use of word spotting, since digital transcription data will already exist for parts of many collections of interest.