InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write
This addresses the gap between pen-and-paper and digital note-taking for users, offering a novel solution with strong generalization.
The paper tackles the problem of converting offline handwriting in photos to digital ink (derendering) by introducing InkSight, which uses vision-language models to read and write, achieving 87% valid tracing and 67% human-like pen trajectory on the HierText dataset.
Digital note-taking is gaining popularity, offering a durable, editable, and easily indexable way of storing notes in a vectorized form, known as digital ink. However, a substantial gap remains between this way of note-taking and traditional pen-and-paper note-taking, a practice that is still favored by a vast majority. Our work InkSight, aims to bridge the gap by empowering physical note-takers to effortlessly convert their work (offline handwriting) to digital ink (online handwriting), a process we refer to as derendering. Prior research on the topic has focused on the geometric properties of images, resulting in limited generalization beyond their training domains. Our approach combines reading and writing priors, allowing training a model in the absence of large amounts of paired samples, which are difficult to obtain. To our knowledge, this is the first work that effectively derenders handwritten text in arbitrary photos with diverse visual characteristics and backgrounds. Furthermore, it generalizes beyond its training domain into simple sketches. Our human evaluation reveals that 87% of the samples produced by our model on the challenging HierText dataset are considered as a valid tracing of the input image and 67% look like a pen trajectory traced by a human.