Surfacing Visualization Mirages
This addresses the problem of deceptive or invalid visualizations for data analysts and visualization practitioners, offering a novel automated detection method that is incremental in applying existing testing techniques to a new domain.
The paper tackles the problem of silent failures in visualizations, termed 'visualization mirages', by adapting metamorphic testing from software testing to automatically detect these issues at the visual encoding stage, showing it can reliably identify mirages across various chart types with minimal prior knowledge.
Dirty data and deceptive design practices can undermine, invert, or invalidate the purported messages of charts and graphs. These failures can arise silently: a conclusion derived from a particular visualization may look plausible unless the analyst looks closer and discovers an issue with the backing data, visual specification, or their own assumptions. We term such silent but significant failures "visualization mirages". We describe a conceptual model of mirages and show how they can be generated at every stage of the visual analytics process. We adapt a methodology from software testing, "metamorphic testing", as a way of automatically surfacing potential mirages at the visual encoding stage of analysis through modifications to the underlying data and chart specification. We show that metamorphic testing can reliably identify mirages across a variety of chart types with relatively little prior knowledge of the data or the domain.