Criteria for Rigor in Visualization Design Study
This addresses the need for methodological rigor in visualization research, particularly for researchers conducting design studies, but it is incremental as it builds on existing discourse from social science and design.
The paper tackles the problem of establishing rigor in visualization design studies by proposing six criteria (INFORMED, REFLEXIVE, ABUNDANT, PLAUSIBLE, RESONANT, TRANSPARENT) from an interpretivist perspective, concluding that the visualization discipline is not yet well-positioned to fully benefit from this approach.
We develop a new perspective on research conducted through visualization design study that emphasizes design as a method of inquiry and the broad range of knowledge-contributions achieved through it as multiple, subjective, and socially constructed. From this interpretivist position we explore the nature of visualization design study and develop six criteria for rigor. We propose that rigor is established and judged according to the extent to which visualization design study research and its reporting are INFORMED, REFLEXIVE, ABUNDANT, PLAUSIBLE, RESONANT, and TRANSPARENT. This perspective and the criteria were constructed through a four-year engagement with the discourse around rigor and the nature of knowledge in social science, information systems, and design. We suggest methods from cognate disciplines that can support visualization researchers in meeting these criteria during the planning, execution, and reporting of design study. Through a series of deliberately provocative questions, we explore implications of this new perspective for design study research in visualization, concluding that as a discipline, visualization is not yet well positioned to embrace, nurture, and fully benefit from a rigorous, interpretivist approach to design study. The perspective and criteria we present are intended to stimulate dialogue and debate around the nature of visualization design study and the broader underpinnings of the discipline.