NL2Scratch: An Executable Benchmark and Evaluation for Block-Based Programming
For researchers in NL2Code and programming education, this work fills a gap by providing an executable benchmark and evaluation metric for block-based languages, revealing limitations of current models that are hidden by lexical metrics.
NL2Scratch introduces a benchmark of 311,648 NL-program pairs for block-based programming in Scratch, along with a semantic alignment metric (SAC). Experiments show that LLMs achieve high lexical similarity (F1 > 0.93) but often fail on semantic alignment, especially for longer examples and operational slots.
Block-based programming environments such as Scratch are widely used in early programming education, yet natural-language-to-code (NL2Code) research has focused primarily on text-based languages. Scratch programs are event-driven, visually compositional, and distributed across concurrent scripts, making conventional NL2Code assumptions and evaluation insufficient. We introduce NL2Scratch, an executable benchmark for natural-language-to-Scratch generation comprising 311,648 parser-valid NL--program pairs, whose program side is extracted from real Scratch projects and paired with semantically aligned NL descriptions. For reliable evaluation beyond surface overlap, we propose Semantic Alignment Consistency (SAC), an interpretable slot-level metric for measuring semantic agreement between descriptions and programs. With SAC, we construct a semantically validated pool of 23,594 examples, and a slot-balanced 800 diagnostic benchmark. Experiments across instruction-tuned and fine-tuned LLMs reveal a notable gap between lexical similarity and semantic alignment: models achieving token-level F1 above 0.93 often fail to attain perfect SAC, particularly on longer examples. Errors concentrate on operational slots like actions, conditions, and numeric arguments, exposing failure modes largely invisible under conventional metrics.