Fidelity of Statistical Reporting in 10 Years of Cyber Security User Studies
This research identifies reporting issues in cyber security user studies, which could undermine validity for practitioners and scientists, though it is incremental as it builds on existing methods from psychology.
The study analyzed the fidelity of statistical reporting in 114 cyber security user studies from 2006-2016, finding that half reported incomplete results, with a slight increase over time and minor venue-based differences.
Studies in socio-technical aspects of security often rely on user studies and statistical inferences on investigated relations to make their case. They, thereby, enable practitioners and scientists alike to judge on the validity and reliability of the research undertaken. To ascertain this capacity, we investigated the reporting fidelity of security user studies. Based on a systematic literature review of $114$ user studies in cyber security from selected venues in the 10 years 2006--2016, we evaluated fidelity of the reporting of $1775$ statistical inferences using the \textsf{R} package \textsf{statcheck}. We conducted a systematic classification of incomplete reporting, reporting inconsistencies and decision errors, leading to multinomial logistic regression (MLR) on the impact of publication venue/year as well as a comparison to a compatible field of psychology. We found that half the cyber security user studies considered reported incomplete results, in stark difference to comparable results in a field of psychology. Our MLR on analysis outcomes yielded a slight increase of likelihood of incomplete tests over time, while SOUPS yielded a few percent greater likelihood to report statistics correctly than other venues. In this study, we offer the first fully quantitative analysis of the state-of-play of socio-technical studies in security. While we highlight the impact and prevalence of incomplete reporting, we also offer fine-grained diagnostics and recommendations on how to respond to the situation.