Describing Console I/O Behavior for Testing Student Submissions in Haskell
This addresses the problem of automated testing and feedback for student programming assignments in Haskell, though it appears incremental as it builds on existing specification language concepts.
The authors developed a formal language for specifying console I/O program behavior, specifically for testing interactive Haskell programs written by students, enabling probabilistic mechanical checking of program behavior against specifications.
We present a small, formal language for specifying the behavior of simple console I/O programs. The design is driven by the concrete application case of testing interactive Haskell programs written by students. Specifications are structurally similar to lexical analysis regular expressions, but are augmented with features like global variables that track state and history of program runs, enabling expression of an interesting range of dynamic behavior. We give a semantics for our specification language based on acceptance of execution traces. From this semantics we derive a definition of the set of all traces valid for a given specification. Sampling that set enables us to mechanically check program behavior against specifications in a probabilistic fashion. Beyond testing, other possible uses of the specification language in an education context include related activities like providing more helpful feedback, generating sample solutions, and even generating random exercise tasks.