Designing Optimal Binary Rating Systems
This work addresses the challenge of improving rating accuracy for online platforms, though it appears incremental as it builds on existing binary feedback models.
The paper tackles the problem of designing optimal binary rating systems for online platforms by formalizing performance as the speed of recovering true item rankings and providing an efficient algorithm to compute the highest-performing system, validated with real-world data from Amazon Mechanical Turk.
Modern online platforms rely on effective rating systems to learn about items. We consider the optimal design of rating systems that collect binary feedback after transactions. We make three contributions. First, we formalize the performance of a rating system as the speed with which it recovers the true underlying ranking on items (in a large deviations sense), accounting for both items' underlying match rates and the platform's preferences. Second, we provide an efficient algorithm to compute the binary feedback system that yields the highest such performance. Finally, we show how this theoretical perspective can be used to empirically design an implementable, approximately optimal rating system, and validate our approach using real-world experimental data collected on Amazon Mechanical Turk.