Evaluating Deception Detection Model Robustness To Linguistic Variation
This work addresses robustness in misinformation detection models, which is important for online platforms, but it is incremental as it builds on existing methods and benchmarks.
The paper tackled the problem of model robustness to linguistic variation in deceptive news detection, finding that character or mixed ensemble models are the most effective defenses and that character perturbation-based attacks are more successful.
With the increasing use of machine-learning driven algorithmic judgements, it is critical to develop models that are robust to evolving or manipulated inputs. We propose an extensive analysis of model robustness against linguistic variation in the setting of deceptive news detection, an important task in the context of misinformation spread online. We consider two prediction tasks and compare three state-of-the-art embeddings to highlight consistent trends in model performance, high confidence misclassifications, and high impact failures. By measuring the effectiveness of adversarial defense strategies and evaluating model susceptibility to adversarial attacks using character- and word-perturbed text, we find that character or mixed ensemble models are the most effective defenses and that character perturbation-based attack tactics are more successful.