Automating Visualization Quality Assessment: a Case Study in Higher Education
This addresses the challenge of efficient and objective assessment for educators and students in higher education, but it is incremental as it builds on existing methods in a specific context.
The study tackled the problem of automating visualization quality assessment in higher education by applying image informatics algorithms to student visualizations, resulting in positive feedback from markers and students, with no concrete numbers provided.
We present a case study in the use of machine+human mixed intelligence for visualization quality assessment, applying automated visualization quality metrics to support the human assessment of data visualizations produced as coursework by students taking higher education courses. A set of image informatics algorithms including edge congestion, visual saliency and colour analysis generate machine analysis of student visualizations. The insight from the image informatics outputs has proved helpful for the marker in assessing the work and is also provided to the students as part of a written report on their work. Student and external reviewer comments suggest that the addition of the image informatics outputs to the standard feedback document was a positive step. We review the ethical challenges of working with assessment data and of automating assessment processes.