A Streaming Analytics Language for Processing Cyber Data
This work addresses the challenge of real-time malicious actor identification in cyber data for network analysts, representing an incremental improvement in domain-specific tools.
The authors tackled the problem of processing large-scale network data for botnet detection by developing a domain-specific language (SAL) and interpreter (SAM) for semi-streaming analytics, achieving an average AUC of 0.87 for botnet detection and scaling to handle 373,000 netflows per second.
We present a domain-specific language called SAL(the Streaming Analytics Language) for processing data in a semi-streaming model. In particular we examine the use case of processing netflow data in order to identify malicious actors within a network. Because of the large volume of data generated from networks, it is often only feasible to process the data with a single pass, utilizing a streaming (O(polylog n) space requirements) or semi-streaming computing model ( O(n polylog n) space requirements). Despite these constraints, we are able to achieve an average of 0.87 for the AUC of the ROC curve for a set of situations dealing with botnet detection. The implementation of an interpreter for SAL, which we call SAM (Streaming Analytics Machine), achieves scaling results that show improved throughput to 61 nodes (976 cores), with an overall rate of 373,000 netflows per second or 32.2 billion per day. SAL provides a succinct way to describe common analyses that allow cyber analysts to find data of interest, and SAM is a scalable interpreter of the language.