GMP: A Benchmark for Content Moderation under Co-occurring Violations and Dynamic Rules
This addresses inconsistent moderation in online platforms, which can lead to harmful content or over-censorship, but is incremental as it focuses on benchmarking rather than solving the underlying AI limitations.
The paper tackles the problem of AI content moderation failing in real-world scenarios with co-occurring violations and dynamic rules, by introducing a benchmark to evaluate generalization, showing current models degrade in performance under these conditions.
Online content moderation is essential for maintaining a healthy digital environment, and reliance on AI for this task continues to grow. Consider a user comment using national stereotypes to insult a politician. This example illustrates two critical challenges in real-world scenarios: (1) Co-occurring Violations, where a single post violates multiple policies (e.g., prejudice and personal attacks); (2) Dynamic rules of moderation, where determination of a violation depends on platform-specific guidelines that evolve across contexts . The intersection of co-occurring harms and dynamically changing rules highlights a core limitation of current AI systems: although large language models (LLMs) are adept at following fixed guidelines, their judgment capabilities degrade when policies are unstable or context-dependent . In practice, such shortcomings lead to inconsistent moderation: either erroneously restricting legitimate expression or allowing harmful content to remain online . This raises a critical question for evaluation: Does high performance on existing static benchmarks truly guarantee robust generalization of AI judgment to real-world scenarios involving co-occurring violations and dynamically changing rules?