Hyperscaling Internet Graph Analysis with D4M on the MIT SuperCloud
This work addresses the problem of processing large-scale network traffic data efficiently for network analysts, though it appears incremental as it applies an existing method (D4M) to a specific domain.
The paper tackled the challenge of analyzing massive network traffic data for anomaly detection by developing a scalable analytics pipeline using D4M on the MIT SuperCloud, achieving a speedup of over 20,000 with only 135 lines of code on 96 hours of Gigabit PCAP data.
Detecting anomalous behavior in network traffic is a major challenge due to the volume and velocity of network traffic. For example, a 10 Gigabit Ethernet connection can generate over 50 MB/s of packet headers. For global network providers, this challenge can be amplified by many orders of magnitude. Development of novel computer network traffic analytics requires: high level programming environments, massive amount of packet capture (PCAP) data, and diverse data products for "at scale" algorithm pipeline development. D4M (Dynamic Distributed Dimensional Data Model) combines the power of sparse linear algebra, associative arrays, parallel processing, and distributed databases (such as SciDB and Apache Accumulo) to provide a scalable data and computation system that addresses the big data problems associated with network analytics development. Combining D4M with the MIT SuperCloud manycore processors and parallel storage system enables network analysts to interactively process massive amounts of data in minutes. To demonstrate these capabilities, we have implemented a representative analytics pipeline in D4M and benchmarked it on 96 hours of Gigabit PCAP data with MIT SuperCloud. The entire pipeline from uncompressing the raw files to database ingest was implemented in 135 lines of D4M code and achieved speedups of over 20,000.