Am I Rare? An Intelligent Summarization Approach for Identifying Hidden Anomalies
This work addresses the challenge of efficiently detecting hidden anomalies in network traffic data, which is a problem for network administrators and security analysts.
This paper proposes INSIDENT, an intelligent summarization approach for identifying hidden anomalies in network traffic data. The method guarantees to preserve the original data distribution in the summarized data, making it suitable for use as a pre-processing step or as an anomaly detection algorithm itself.
Monitoring network traffic data to detect any hidden patterns of anomalies is a challenging and time-consuming task that requires high computing resources. To this end, an appropriate summarization technique is of great importance, where it can be a substitute for the original data. However, the summarized data is under the threat of removing anomalies. Therefore, it is vital to create a summary that can reflect the same pattern as the original data. Therefore, in this paper, we propose an INtelligent Summarization approach for IDENTifying hidden anomalies, called INSIDENT. The proposed approach guarantees to keep the original data distribution in summarized data. Our approach is a clustering-based algorithm that dynamically maps original feature space to a new feature space by locally weighting features in each cluster. Therefore, in new feature space, similar samples are closer, and consequently, outliers are more detectable. Besides, selecting representatives based on cluster size keeps the same distribution as the original data in summarized data. INSIDENT can be used both as the preprocess approach before performing anomaly detection algorithms and anomaly detection algorithm. The experimental results on benchmark datasets prove a summary of the data can be a substitute for original data in the anomaly detection task.