Self-Supervised Losses for One-Class Textual Anomaly Detection
This provides a simpler, more effective method for one-class textual anomaly detection, addressing the need for supervisory signals or complex architectures, though it is incremental as it builds on existing self-supervised learning techniques.
The paper tackles the problem of anomaly detection in text by fine-tuning Transformers on inlier data with self-supervised objectives and using the losses as anomaly scores, resulting in improved AUROC scores by 11.6% for semantic anomalies and 22.8% for syntactic anomalies on average.
Current deep learning methods for anomaly detection in text rely on supervisory signals in inliers that may be unobtainable or bespoke architectures that are difficult to tune. We study a simpler alternative: fine-tuning Transformers on the inlier data with self-supervised objectives and using the losses as an anomaly score. Overall, the self-supervision approach outperforms other methods under various anomaly detection scenarios, improving the AUROC score on semantic anomalies by 11.6% and on syntactic anomalies by 22.8% on average. Additionally, the optimal objective and resultant learnt representation depend on the type of downstream anomaly. The separability of anomalies and inliers signals that a representation is more effective for detecting semantic anomalies, whilst the presence of narrow feature directions signals a representation that is effective for detecting syntactic anomalies.