Measuring and Improving Model-Moderator Collaboration using Uncertainty Estimation
This work addresses the challenge of designing effective human-AI collaboration systems for content moderation, with incremental improvements in metrics and strategies.
The study tackled the problem of optimizing human-model collaboration in content moderation by introducing metrics to evaluate system performance under human capacity constraints and conducting a benchmark of uncertainty-based review strategies. The results showed that uncertainty-based strategies consistently outperformed toxicity score-based ones, significantly improving overall system performance.
Content moderation is often performed by a collaboration between humans and machine learning models. However, it is not well understood how to design the collaborative process so as to maximize the combined moderator-model system performance. This work presents a rigorous study of this problem, focusing on an approach that incorporates model uncertainty into the collaborative process. First, we introduce principled metrics to describe the performance of the collaborative system under capacity constraints on the human moderator, quantifying how efficiently the combined system utilizes human decisions. Using these metrics, we conduct a large benchmark study evaluating the performance of state-of-the-art uncertainty models under different collaborative review strategies. We find that an uncertainty-based strategy consistently outperforms the widely used strategy based on toxicity scores, and moreover that the choice of review strategy drastically changes the overall system performance. Our results demonstrate the importance of rigorous metrics for understanding and developing effective moderator-model systems for content moderation, as well as the utility of uncertainty estimation in this domain.