Can We Improve Educational Diagram Generation with In-Context Examples? Not if a Hallucination Spoils the Bunch
This addresses concerns among educators and students about AI-generated educational materials, though it is incremental as it builds on existing in-context learning methods.
The study tackled the problem of low-quality educational diagram generation by AI by introducing a method based on Rhetorical Structure Theory to reduce factual hallucination and improve faithfulness to context, with preliminary results showing decreased hallucination rates but variable diagram quality due to LLM stochasticity.
Generative artificial intelligence (AI) has found a widespread use in computing education; at the same time, quality of generated materials raises concerns among educators and students. This study addresses this issue by introducing a novel method for diagram code generation with in-context examples based on the Rhetorical Structure Theory (RST), which aims to improve diagram generation by aligning models' output with user expectations. Our approach is evaluated by computer science educators, who assessed 150 diagrams generated with large language models (LLMs) for logical organization, connectivity, layout aesthetic, and AI hallucination. The assessment dataset is additionally investigated for its utility in automated diagram evaluation. The preliminary results suggest that our method decreases the rate of factual hallucination and improves diagram faithfulness to provided context; however, due to LLMs' stochasticity, the quality of the generated diagrams varies. Additionally, we present an in-depth analysis and discussion on the connection between AI hallucination and the quality of generated diagrams, which reveals that text contexts of higher complexity lead to higher rates of hallucination and LLMs often fail to detect mistakes in their output.