The Validity, Generalizability and Feasibility of Summative Evaluation Methods in Visual Analytics
This work addresses the problem of selecting and improving evaluation methods for researchers and practitioners in Visual Analytics, though it is incremental as it builds on existing surveys and taxonomies.
The paper analyzes summative evaluation methods in Visual Analytics (VA) to address confusion about their proofing capabilities and limited discussion on evaluation processes, proposing a new metric called summative quality for comparing methods and making recommendations based on this metric in the VA domain.
Many evaluation methods have been used to assess the usefulness of Visual Analytics (VA) solutions. These methods stem from a variety of origins with different assumptions and goals, which cause confusion about their proofing capabilities. Moreover, the lack of discussion about the evaluation processes may limit our potential to develop new evaluation methods specialized for VA. In this paper, we present an analysis of evaluation methods that have been used to summatively evaluate VA solutions. We provide a survey and taxonomy of the evaluation methods that have appeared in the VAST literature in the past two years. We then analyze these methods in terms of validity and generalizability of their findings, as well as the feasibility of using them. We propose a new metric called summative quality to compare evaluation methods according to their ability to prove usefulness, and make recommendations for selecting evaluation methods based on their summative quality in the VA domain.