Text to Automata Diagrams: Comparing TikZ Code Generation with Direct Image Synthesis
This research addresses the problem of automating the processing and grading of student-drawn diagrams for computer science educators, offering an incremental step towards automated feedback and accessible materials.
This study investigates the ability of vision-language models (VLMs) and large language models (LLMs) to convert scanned, student-drawn automata diagrams into TikZ code. It found that VLM-generated descriptions were often incorrect, but human correction significantly improved the quality of the final TikZ diagrams.
Diagrams are widely used in teaching computer science courses. They are useful in subjects such as automata and formal languages, data structures, etc. These diagrams, often drawn by students during exams or assignments, vary in structure, layout, and correctness. This study examines whether current vision-language and large language models can process such diagrams and produce accurate textual and digital representations. In this study, scanned student-drawn diagrams are used as input. Then, textual descriptions are generated from these images using a vision-language model. The descriptions are checked and revised by human reviewers to make them accurate. Both the generated and the revised descriptions are then fed to a large language model to generate TikZ code. The resulting diagrams are compiled and then evaluated against the original scanned diagrams. We found descriptions generated directly from images using vision-language models are often incorrect and human correction can substantially improve the quality of vision language model generated descriptions. This research can help computer science education by paving the way for automated grading and feedback and creating more accessible instructional materials.