Exploring Outliers in Crowdsourced Ranking for QoE
This provides incremental tools for robust QoE evaluation in crowdsourcing, addressing a domain-specific problem for researchers and practitioners in quality assessment.
The paper tackled outlier detection in crowdsourced Quality of Experience (QoE) ranking by proposing simple, fast algorithms based on nonconvex optimization, achieving similar performance to Huber-LASSO with up to 90 times speed-up.
Outlier detection is a crucial part of robust evaluation for crowdsourceable assessment of Quality of Experience (QoE) and has attracted much attention in recent years. In this paper, we propose some simple and fast algorithms for outlier detection and robust QoE evaluation based on the nonconvex optimization principle. Several iterative procedures are designed with or without knowing the number of outliers in samples. Theoretical analysis is given to show that such procedures can reach statistically good estimates under mild conditions. Finally, experimental results with simulated and real-world crowdsourcing datasets show that the proposed algorithms could produce similar performance to Huber-LASSO approach in robust ranking, yet with nearly 8 or 90 times speed-up, without or with a prior knowledge on the sparsity size of outliers, respectively. Therefore the proposed methodology provides us a set of helpful tools for robust QoE evaluation with crowdsourcing data.