Comparing Video Based Shoulder Surfing with Live Simulation
This work addresses the reliability of security testing methods for researchers, highlighting that video simulations may provide a baseline but can be misleading for certain authentication methods like PINs.
The study compared video-based shoulder surfing simulations with live attacks, finding that video simulations accurately reflect attacker success rates for Android graphical patterns but significantly underestimate threats for PINs, with live attackers performing up to 1.9 times better.
We analyze the claims that video recreations of shoulder surfing attacks offer a suitable alternative and a baseline, as compared to evaluation in a live setting. We recreated a subset of the factors of a prior video-simulation experiment conducted by Aviv et al. (ACSAC 2017), and model the same scenario using live participants ($n=36$) instead (i.e., the victim and attacker were both present). The live experiment confirmed that for Android's graphical patterns video simulation is consistent with the live setting for attacker success rates. However, both 4- and 6-digit PINs demonstrate statistically significant differences in attacker performance, with live attackers performing as much 1.9x better than in the video simulation. The security benefits gained from removing feedback lines in Android's graphical patterns are also greatly diminished in the live setting, particularly under multiple attacker observations, but overall, the data suggests that video recreations can provide a suitable baseline measure for attacker success rate. However, we caution that researchers should consider that these baselines may greatly underestimate the threat of an attacker in live settings.