When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition
This work addresses the challenge of recognizing complex or long handwritten mathematical expressions for applications in education and document analysis, representing an incremental improvement over existing methods.
The paper tackles the problem of inaccurate attention in encoder-decoder networks for handwritten mathematical expression recognition (HMER) by proposing a Counting-Aware Network (CAN) that jointly optimizes HMER and symbol counting, achieving state-of-the-art performance on benchmark datasets with marginal extra time cost.
Recently, most handwritten mathematical expression recognition (HMER) methods adopt the encoder-decoder networks, which directly predict the markup sequences from formula images with the attention mechanism. However, such methods may fail to accurately read formulas with complicated structure or generate long markup sequences, as the attention results are often inaccurate due to the large variance of writing styles or spatial layouts. To alleviate this problem, we propose an unconventional network for HMER named Counting-Aware Network (CAN), which jointly optimizes two tasks: HMER and symbol counting. Specifically, we design a weakly-supervised counting module that can predict the number of each symbol class without the symbol-level position annotations, and then plug it into a typical attention-based encoder-decoder model for HMER. Experiments on the benchmark datasets for HMER validate that both joint optimization and counting results are beneficial for correcting the prediction errors of encoder-decoder models, and CAN consistently outperforms the state-of-the-art methods. In particular, compared with an encoder-decoder model for HMER, the extra time cost caused by the proposed counting module is marginal. The source code is available at https://github.com/LBH1024/CAN.