An Aligned Rank Transform Procedure for Multifactor Contrast Tests
This addresses a specific issue for researchers in HCI and statistics dealing with nonconforming data, though it is incremental as it builds on the existing ART paradigm.
The paper tackled the problem of incorrect contrast tests in nonparametric analysis of multifactor HCI experiments using the Aligned Rank Transform (ART), resulting in a new algorithm called ART-C that avoids inflating Type I error rates and shows higher statistical power than several existing tests, validated on 72,000 data sets.
Data from multifactor HCI experiments often violates the normality assumption of parametric tests (i.e., nonconforming data). The Aligned Rank Transform (ART) is a popular nonparametric analysis technique that can find main and interaction effects in nonconforming data, but leads to incorrect results when used to conduct contrast tests. We created a new algorithm called ART-C for conducting contrasts within the ART paradigm and validated it on 72,000 data sets. Our results indicate that ART-C does not inflate Type I error rates, unlike contrasts based on ART, and that ART-C has more statistical power than a t-test, Mann-Whitney U test, Wilcoxon signed-rank test, and ART. We also extended a tool called ARTool with our ART-C algorithm for both Windows and R. Our validation had some limitations (e.g., only six distribution types, no mixed factorial designs, no random slopes), and data drawn from Cauchy distributions should not be analyzed with ART-C.