State Frequency Estimation for Anomaly Detection
This addresses the issue of adversaries evading detection in network security by producing common-looking traces, though it appears incremental as it builds on existing state machine methods.
The paper tackles the problem of anomaly detection in network flows by proposing SEQUENT, an unsupervised method that dynamically adapts anomaly scores based on state visit frequencies in state machines, and it shows promising results on three public NetFlow datasets.
Many works have studied the efficacy of state machines for detecting anomalies within NetFlows. These works typically learn a model from unlabeled data and compute anomaly scores for arbitrary traces based on their likelihood of occurrence or how well they fit within the model. However, these methods do not dynamically adapt their scores based on the traces seen at test time. This becomes a problem when an adversary produces seemingly common traces in their attack, causing the model to miss the detection by assigning low anomaly scores. We propose SEQUENT, a new unsupervised approach that uses the state visit frequency of a state machine to adapt its scoring dynamically for anomaly detection. SEQUENT subsequently uses the scores to generate root causes for anomalies. These allow the grouping of alarms and simplify the analysis of anomalies. We evaluate SEQUENT's effectiveness in detecting network anomalies on three publicly available NetFlow datasets and compare its performance against various existing unsupervised anomaly detection methods. Our evaluation shows promising results for using the state visit frequency of a state machine to detect network anomalies.