Teaching Machines to Code: Neural Markup Generation with Visual Attention
This work addresses the challenge of automatically converting visual math formulas into editable code, which is incremental as it builds on existing sequence modeling techniques but applies them to a specific domain with improved performance.
The authors tackled the problem of generating LaTeX markup from images of math formulas using a neural transducer model with visual attention, achieving a BLEU score of 89% and producing code over 150 words long, which improves upon the previous state-of-the-art for the Im2Latex problem.
We present a neural transducer model with visual attention that learns to generate LaTeX markup of a real-world math formula given its image. Applying sequence modeling and transduction techniques that have been very successful across modalities such as natural language, image, handwriting, speech and audio; we construct an image-to-markup model that learns to produce syntactically and semantically correct LaTeX markup code over 150 words long and achieves a BLEU score of 89%; improving upon the previous state-of-art for the Im2Latex problem. We also demonstrate with heat-map visualization how attention helps in interpreting the model and can pinpoint (detect and localize) symbols on the image accurately despite having been trained without any bounding box data.