NLP-ADBench: NLP Anomaly Detection Benchmark
This provides a unified framework for NLP anomaly detection, addressing a gap for researchers and practitioners in fields like fraud detection and content moderation, though it is incremental as it builds on existing methods.
The authors tackled the lack of comprehensive benchmarks for anomaly detection in natural language processing by introducing NLP-ADBench, which includes eight datasets and 19 algorithms, showing that two-step methods with transformer-based embeddings outperform end-to-end approaches, with OpenAI embeddings beating BERT.
Anomaly detection (AD) is an important machine learning task with applications in fraud detection, content moderation, and user behavior analysis. However, AD is relatively understudied in a natural language processing (NLP) context, limiting its effectiveness in detecting harmful content, phishing attempts, and spam reviews. We introduce NLP-ADBench, the most comprehensive NLP anomaly detection (NLP-AD) benchmark to date, which includes eight curated datasets and 19 state-of-the-art algorithms. These span 3 end-to-end methods and 16 two-step approaches that adapt classical, non-AD methods to language embeddings from BERT and OpenAI. Our empirical results show that no single model dominates across all datasets, indicating a need for automated model selection. Moreover, two-step methods with transformer-based embeddings consistently outperform specialized end-to-end approaches, with OpenAI embeddings outperforming those of BERT. We release NLP-ADBench at https://github.com/USC-FORTIS/NLP-ADBench, providing a unified framework for NLP-AD and supporting future investigations.