InkFM: A Foundational Model for Full-Page Online Handwritten Note Understanding
This addresses the need for efficient workflow optimization in tablet and stylus note-taking by providing a versatile model for handwritten content understanding.
The paper tackles the problem of interpreting full pages of handwritten digital notes by introducing InkFM, a foundational model that unifies text recognition in 28 scripts, mathematical expression recognition, and page segmentation, achieving state-of-the-art text line segmentation and competitive performance on multiple datasets.
Tablets and styluses are increasingly popular for taking notes. To optimize this experience and ensure a smooth and efficient workflow, it's important to develop methods for accurately interpreting and understanding the content of handwritten digital notes. We introduce a foundational model called InkFM for analyzing full pages of handwritten content. Trained on a diverse mixture of tasks, this model offers a unique combination of capabilities: recognizing text in 28 different scripts, mathematical expressions recognition, and segmenting pages into distinct elements like text and drawings. Our results demonstrate that these tasks can be effectively unified within a single model, achieving SoTA text line segmentation out-of-the-box quality surpassing public baselines like docTR. Fine- or LoRA-tuning our base model on public datasets further improves the quality of page segmentation, achieves state-of the art text recognition (DeepWriting, CASIA, SCUT, and Mathwriting datasets) and sketch classification (QuickDraw). This adaptability of InkFM provides a powerful starting point for developing applications with handwritten input.