Beyond Advocacy: A Design Space for Replication-Related Studies
For researchers in visualization and HCI, this provides a structured way to design and analyze replication studies, though it is an incremental conceptual contribution.
The paper introduces a multi-dimensional design space framework for categorizing and planning replication studies, treating replication as a pairwise comparison problem with four practical dimensions across three comparison levels.
The importance of replication is often discussed and advocated -- not only in the domains of visualization and HCI, but in all scientific areas. When replicating a study, design decisions need to be made with regards which aspects of the original study will remain the same and which will be altered. We present a supporting multi-dimensional design space framework within which such decisions can be identified, categorized, compared and analyzed. The framework treats replication experimental design as a pairwise comparison problem, and represents the design by four practical dimensions defined by three comparison levels. The design space is therefore a framework that can be used for both retrospective characterization and prospective planning. We provide worked examples, and relate our framework to other attempts at describing the scope of replication studies.