Integrated Clustering and Anomaly Detection (INCAD) for Streaming Data (Revised)
This addresses the challenge of handling streaming data for anomaly detection in domains where thresholds and cluster numbers are unknown, though it appears incremental as it builds on existing clustering-based methods.
The paper tackles the problem of clustering and anomaly detection in streaming data without needing predefined thresholds or cluster counts, resulting in a method that simultaneously performs probabilistic anomaly detection and clustering to improve reliability of normal vs. abnormal behavior definitions.
Most current clustering based anomaly detection methods use scoring schema and thresholds to classify anomalies. These methods are often tailored to target specific data sets with "known" number of clusters. The paper provides a streaming clustering and anomaly detection algorithm that does not require strict arbitrary thresholds on the anomaly scores or knowledge of the number of clusters while performing probabilistic anomaly detection and clustering simultaneously. This ensures that the cluster formation is not impacted by the presence of anomalous data, thereby leading to more reliable definition of "normal vs abnormal" behavior. The motivations behind developing the INCAD model and the path that leads to the streaming model is discussed.