What Shapes Participant Data Quality? A Scoping Review and Case Study of Crowdsourced Webcam Eye Tracking in AI Interviews
For HCI and behavioral science researchers using crowdsourced eye tracking, this paper identifies current quality issues and provides predictive insights to enhance data reliability.
This paper reviews crowdsourced webcam eye tracking practices (2011-2025), finding fragmented reporting and no quality benchmarks. A case study (N=205) using RealEye shows that fixation count, session duration, and OS predict data quality, leading to recommendations for improving reliability.
Webcam-based eye tracking is a cost-effective, scalable method for remote research that effectively reaches broader populations. However, uncontrolled environments and hardware diversity lead to inconsistent data quality in crowdsourcing. To assess current practices, we conducted a scoping review of crowdsourced eye-tracking from 2011-2025. The review confirms fragmented reporting and a lack of established quality benchmarks. To address this lack of predictive insight, we conducted a case study on AI fairness interviews (N=205) using the RealEye platform. Applying Ordered Logistic Regression (OLR) to the platform quality metric, we found that behavioral and technical factors significantly predict data quality. Specifically, within the RealEye platform, higher fixation counts, shorter sessions, and operating system choice yield significantly higher quality grades. Based on this review and platform-specific predictive insights, we provide actionable recommendations to enhance the reliability, transparency, and replicability of future crowdsourced webcam eye tracking in HCI and behavioral science.