Not-in-Perspective: Towards Shielding Google's Perspective API Against Adversarial Negation Attacks
This work addresses the vulnerability of automated toxicity detection systems used in social media moderation to adversarial attacks, offering a hybrid solution that enhances robustness for platforms like Google's Perspective API.
The paper tackles the problem of adversarial negation attacks on toxicity detection systems by introducing a formal reasoning wrapper that acts as pre- and post-processing steps, resulting in significant improvements in accuracy and efficacy against such attacks.
The rise of cyberbullying in social media platforms involving toxic comments has escalated the need for effective ways to monitor and moderate online interactions. Existing solutions of automated toxicity detection systems, are based on a machine or deep learning algorithms. However, statistics-based solutions are generally prone to adversarial attacks that contain logic based modifications such as negation in phrases and sentences. In that regard, we present a set of formal reasoning-based methodologies that wrap around existing machine learning toxicity detection systems. Acting as both pre-processing and post-processing steps, our formal reasoning wrapper helps alleviating the negation attack problems and significantly improves the accuracy and efficacy of toxicity scoring. We evaluate different variations of our wrapper on multiple machine learning models against a negation adversarial dataset. Experimental results highlight the improvement of hybrid (formal reasoning and machine-learning) methods against various purely statistical solutions.