Ensuring Reliable Participation in Subjective Video Quality Tests Across Platforms
This addresses the challenge of ensuring accurate and cost-effective video quality assessment for streaming and communication services, but it is incremental as it builds on existing crowdsourcing methods.
The paper tackled the problem of unreliable submissions in crowdsourced subjective video quality tests, which bias results, by proposing detectors for remote-desktop users and comparing two platforms on susceptibility and mitigation, finding that one platform showed higher reliability under realistic conditions.
Subjective video quality assessment (VQA) is the gold standard for measuring end-user experience across communication, streaming, and UGC pipelines. Beyond high-validity lab studies, crowdsourcing offers accurate, reliable, faster, and cheaper evaluation-but suffers from unreliable submissions by workers who ignore instructions or game rewards. Recent tests reveal sophisticated exploits of video metadata and rising use of remote-desktop (RD) connections, both of which bias results. We propose objective and subjective detectors for RD users and compare two mainstream crowdsourcing platforms on their susceptibility and mitigation under realistic test conditions and task designs.