SemiHMER: Semi-supervised Handwritten Mathematical Expression Recognition using pseudo-labels
This work addresses the problem of handwritten mathematical expression recognition, which is significant for individuals and organizations working with mathematical notations, particularly in fields like education, research, and engineering, and presents an incremental improvement over existing methods.
The authors tackled the problem of handwritten mathematical expression recognition and achieved an average accuracy increase of 5.47%, 4.87%, and 5.25% on CROHME14, CROHME16, and CROHME19 datasets, respectively. This was accomplished through their proposed SemiHMER framework, which utilizes pseudo-labels and consistency regularization.
In this paper, we study semi-supervised Handwritten Mathematical Expression Recognition (HMER) via exploring both labeled data and extra unlabeled data. We propose a novel consistency regularization framework, termed SemiHMER, which introduces dual-branch semi-supervised learning. Specifically, we enforce consistency between the two networks for the same input image. The pseudo-label, generated by one perturbed recognition network, is utilized to supervise the other network using the standard cross-entropy loss. The SemiHMER consistency encourages high similarity between the predictions of the two perturbed networks for the same input image and expands the training data by leveraging unlabeled data with pseudo-labels. We further introduce a weak-to-strong strategy by applying different levels of augmentation to each branch, effectively expanding the training data and enhancing the quality of network training. Additionally, we propose a novel module, the Global Dynamic Counting Module (GDCM), to enhance the performance of the HMER decoder by alleviating recognition inaccuracies in long-distance formula recognition and reducing the occurrence of repeated characters. The experimental results demonstrate that our work achieves significant performance improvements, with an average accuracy increase of 5.47% on CROHME14, 4.87% on CROHME16, and 5.25% on CROHME19, compared to our baselines.