How Far Should We Need to Go : Evaluate Provenance-based Intrusion Detection Systems in Industrial Scenarios
This work addresses the gap in evaluating intrusion detection systems for industrial cybersecurity, highlighting practical limitations and offering incremental improvements.
The paper systematically evaluated five state-of-the-art provenance-based intrusion detection systems (PIDSes) in industrial scenarios, revealing challenges such as poor portability, low detection performance against real-world attacks, and high false positive rates, and proposed a method that reduces manual effort by 2/3 to mitigate false positives.
Provenance-based Intrusion Detection Systems (PIDSes) have been widely used to detect Advanced Persistent Threats (APTs). Although many studies achieve high performance in the evaluations of their original papers, their performance in industrial scenarios remains unclear. To fill this gap, we conduct the first systematic evaluation and analysis of PIDSes in industrial scenarios. We first analyze the differences between the data from DARPA datasets and that collected in industrial scenarios, identifying three main new characteristics in industry: heterogeneous multi-source inputs, more powerful attackers, and increasing benign activity complexity. We then build several datasets to evaluate five state-of-the-art PIDSes. The evaluation results reveal challenges for existing PIDSes, including poor portability across different hosts and platforms, low detection performance against real-world attacks, and high false positive rates with ever-changing benign activities. Based on the evaluation results and our industrial practices, we provide several insights to solve or explain the above problems. For example, we propose a method to mitigate the high false positives, which reduces manual effort by 2/3. Finally, we propose several research suggestions to improve PIDSes.